Exiting: error loading config file: config file ("/etc/apm-server/apm-server.yml") must be owned by the user identifier (uid=0) or root

[meiyuan]

APM Server version: 7.0.0.

APM Agent language and version: Node.js

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
https://artifacts.elastic.co/downloads/apm-server/apm-server-7.0.0-x86_64.rpm

Is there anything special in your setup?
We use ansible playbook to install and config APM;

// this would copy the apm-server.yml ot the right plalce
- name: Deploy /etc/apm-server/apm-server.yml config
  template: 
    src: apm-server.yml.j2
    dest: /etc/apm-server/apm-server.yml
  become: true
  
// We want to start the service, `become: true` means become `root` user
- name: Start APM-Server Service
  systemd:
    state: started
    name: apm-server.service
  become: true

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

[deploy@ip-192-168-0-141 ~]$ systemctl status apm-server
● apm-server.service - Elastic APM Server
   Loaded: loaded (/usr/lib/systemd/system/apm-server.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Thu 2019-05-02 21:23:57 UTC; 3 days ago
     Docs: https://www.elastic.co/solutions/apm
  Process: 5707 ExecStart=/usr/share/apm-server/bin/apm-server $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 5707 (code=exited, status=1/FAILURE)

May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: apm-server.service: main process exited, code=exited, status=1/FAILURE
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: Unit apm-server.service entered failed state.
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: apm-server.service failed.
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: apm-server.service holdoff time over, scheduling restart.
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: start request repeated too quickly for apm-server.service
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: Failed to start Elastic APM Server.
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: Unit apm-server.service entered failed state.
May 02 21:23:57 ip-192-168-0-141.ec2.internal systemd[1]: apm-server.service failed.

Provide logs and/or server output (if relevant):
Exiting: error loading config file: config file ("/etc/apm-server/apm-server.yml") must be owned by the user identifier (uid=0) or root

Other comments:

1. The /etc/apm-server/apm-server.yml is owned by the apm-server user.
2. https://github.com/elastic/apm-server/blob/6.3/_beats/dev-tools/packer/platforms/debian/systemd.j2
   This is an old systemd file. I couldn't find the v7.0.0.
   How can I run this service with user apm-server without changing the file permission to root?

[gil (Gil Raphaelli)]

Hello @meiyuan, sorry to see you've run into this issue. This has been fixed in 7.0.1, please upgrade if possible.

If you are unable to upgrade, in 7.0.0 you can patch the service file adding User and Group to work around the problem.

[meiyuan]

Hi Gil.
Yeah, I updated it to 7.0.1 and I was able to pass that error.
Just out of curiosity, how would I patch the service file? Because when the service fails to load, there is no file in /etc/systemd/system. Should I just create my own service file from scratch? Is there a template for us to use?

[meiyuan]

I read your comment in that git issue. Still not very clear on how we would patch it since there is no such file.
Thank you very much.

[gil (Gil Raphaelli)]

The file to patch is /usr/lib/systemd/system/apm-server.service, that should be created when installing the rpm. This should do the job:

sudo patch /usr/lib/systemd/system/apm-server.service << EOF
--- apm-server.service.orig 0000-00-00 00:00:00.000000000 +0000
+++ apm-server.service0000-00-00 00:00:00.000000000 +0000
@@ -5,6 +5,8 @@
 After=network-online.target
 
 [Service]
+User=apm-server
+Group=apm-server
 Environment="BEAT_LOG_OPTS=-e"
 Environment="BEAT_CONFIG_OPTS=-c /etc/apm-server/apm-server.yml"
 Environment="BEAT_PATH_OPTS=-path.home /usr/share/apm-server -path.config /etc/apm-server -path.data /var/lib/apm-server -path.logs /var/log/apm-server"
EOF

# reload
sudo systemctl daemon-reload