Can't get delete by query to actually delete

I don't know. If you have a way to reproduce the problem with a sample script that creates a test index, index a document, run the delete by query and run the reindex, that could help.