I have an automated search and insert program based on elasticsearch-dsl.py and scrapy using the python elasticsearch pipeline for bulk insert into es.
It runs flawlessly when I start it as root, but when I try to run as cronjob, the scrapy section works but the bulk insert fails to insert with no error messages. I have tried numerous cron job ideas but to no avail. The entire process finishes but there is no actual data inserted.
I have been struggling with this for some time, does anyone have any ideas ? I tried stackoverflow.com with no response.