Python elasticsearch-dsl bulk insert will not work as a cron job otherwise it works fine

Elastic Stack Elasticsearch
1

Hi --

I have an automated search and insert program based on elasticsearch-dsl.py and scrapy using the python elasticsearch pipeline for bulk insert into es.

It runs flawlessly when I start it as root, but when I try to run as cronjob, the scrapy section works but the bulk insert fails to insert with no error messages. I have tried numerous cron job ideas but to no avail. The entire process finishes but there is no actual data inserted.

I have been struggling with this for some time, does anyone have any ideas ? I tried stackoverflow.com with no response.

Thank you

2

Ok .. I figured it out -- I was using absolute paths to the python scripts but failed to remember that cron jobs are started in the home directory, so there was not path to the settings file. a simple cd /home/midir && ./mypython.py worked ..

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.