Ingesting documents (pdf, word, .txt) to elasticsearch

This is wrong. You will end up with a lot of processes running in parallel.

You should use:

* * * * * bash /fscrawler.sh jobs --restart --loop 1

Well. If you don't use --restart option, any time you launch again fscrawler, it should detect files which have been removed in the meantime.

If you use --restart I think that it will not detect files removal. But that's a guess as I never tested that.

loop has no effect on detection. It's only there to exit after a given number of runs.