I wanted to install ElastAlert, an alerting tool written in python2, inside of a pipenv in an attempt to keep applications and their requirements/ dependencies from stepping on one another while running with the lowest possible permissions. To accomplish this, I generated a python2 virtual environment with pipenv --two and proceeded to install ElastAlert within it. I configured it to work with my existing elastic stack and success! ElastAlert successfully alerts when running from the command line.
I want this thing to run on start and survive the odd crash so I put together a systemd module to support this my needs and easy orchestration. The *.service file is as follows:
[Unit]
Description=ElastAlert
Documentation=https://elastalert.readthedocs.io
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
Restart=always
User=easervice
Group=easervice
WorkingDirectory=/home/easervice
ExecStart=/home/easervice/.local/bin/pipenv run /home/easervice/.local/bin/python /home/easervice/.local/share/virtualenvs/easervice-r3wr-03C/bin/elastalert --verbose --config /etc/elastalert/config.yaml
[Install]
WantedBy=multi-user.target
The logs are:
Jun 16 21:56:30 host systemd[27895]: elastalert.service: Failed at step CHDIR spawning /usr/bin/elastalert: No such file or directory
Jun 16 21:56:30 host systemd[1]: Started elastalert.
Jun 16 21:56:30 host systemd[1]: elastalert.service: Main process exited, code=exited, status=200/CHDIR
Jun 16 21:56:30 host systemd[1]: elastalert.service: Unit entered failed state.
Jun 16 21:56:30 host systemd[1]: elastalert.service: Failed with result 'exit-code'.
The executable exists in the same space within the venv as outside.
Launching subshell in virtual environment…
easervice@host:~$ . /home/easervice/.local/share/virtualenvs/easervice-r3wr-03C/bin/activate
(easervice) easervice@host:~$ which elastalert
/home/easervice/.local/share/virtualenvs/easervice-r3wr-03C/bin/elastalert
(easervice) easervice@host:~$
200/CHDIR generally stems from 2 things based on my reading:
- Bad permissions
- File does not exist
Things I have tried to get ExecStart to complete:
- define different pythons by absolute path
- Symlinking my elastalert executable into
/usr/bin/elastalert - changing user to root
- confirmed that the
ExecStartcommand works from botheaserviceandroot
Sources I used to get started:
- http://www.robpeck.com/2017/11/using-pipenv-with-systemd/#.XQHp0YhKgjb
- https://serverfault.com/questions/821575/systemd-run-a-python-script-at-startup-virtualenv
I have run out of ideas on what to weak and have not found great success via my google foo. Any idea how to make this go?
UPDATE 1:
Attempted to add an environment variable to direct PYTHONPATH:
Environment="PYTHONPATH=/home/elastalert/.local/lib/python2.7/site-packages"
Resulted in the same error.
Attempted letting the pipenv handle paths by removing hardcoded paths and received the same log error output. The config used is as follows:
Description=ElastAlert
Documentation=https://elastalert.readthedocs.io
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
Restart=always
User=easervice
Group=easervice
Environment="PYTHONPATH=/home/elastalert/.local/lib/python2.7/site-packages"
WorkingDirectory=/home/easervice
ExecStart=/home/easervice/.local/bin/pipenv run elastalert --verbose --config /etc/elastalert/config.yaml
[Install]
WantedBy=multi-user.target
Out of curiosity, I also installed it through pip straight to the user and got the same log output, despite it ALSO working from the command line. The file used for that attempt is as follows:
[Unit]
Description=ElastAlert
Documentation=https://elastalert.readthedocs.io
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
Restart=always
User=easervice
Group=easervice
ExecStart=/home/easervice/.local/bin/elastalert --verbose --config /etc/elastalert/config.yaml
[Install]
WantedBy=multi-user.target
Update 2:
I attempted running:
ExecStart=/bin/sh -c '/home/easervice/.local/bin/elastalert --verbose --config /etc/elastalert/config.yaml'
and
ExecStart=/bin/sh -c '/home/easervice/.local/bin/pipenv run elastalert --verbose --config /etc/elastalert/config.yaml'
I received the exact same error. flummoxed