Right - got this search engine fixed. Two problems:

1. I was indexing with a StopAnalyzer and then searching with a StandardAnalyzer.

2. I was wrapping the query object returned from the QueryParser in my own PermissibleUrlQuery object, which does something broken (and deletes half the results).

(Lucene was not to blame: its incremental indexing is working fine :-)