APM Server breaking on startup

Hi all,

having an issue starting an apm-server on an ubuntu vm. See details below.

Kibana version: 7.4.0

Elasticsearch version: 7.4.0

APM Server version: 7.4.0

APM Agent language and version: N/A

Browser version: Version 83.0.4103.97 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: deb

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. - None of the above

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

When starting the apm-server service (service apm-server start) the server starts up but then exits straight away with a generic error on a missing path.
Can't really understand which path it's referring to. I didn't change the path config in apm-server.yml so using defaults.

Provide logs and/or server output (if relevant): apm-server debug logs below

2020-06-21T22:05:44.269Z	INFO	instance/beat.go:607	Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]
2020-06-21T22:05:44.269Z	DEBUG	[beat]	instance/beat.go:659	Beat metadata path: /var/lib/apm-server/meta.json
2020-06-21T22:05:44.269Z	INFO	instance/beat.go:615	Beat ID: 7d3ea0e7-866d-4e67-bf4e-54a4958a72c2
2020-06-21T22:05:44.270Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-06-21T22:05:44.270Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-06-21T22:05:44.270Z	INFO	[beat]	instance/beat.go:903	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/apm-server", "data": "/var/lib/apm-server", "home": "/usr/share/apm-server", "logs": "/var/log/apm-server"}, "type": "apm-server", "uuid": "7d3ea0e7-866d-4e67-bf4e-54a4958a72c2"}}}
2020-06-21T22:05:44.270Z	INFO	[beat]	instance/beat.go:912	Build info	{"system_info": {"build": {"commit": "971d864356e4438bf4a799a1fa052cfd0ce680b4", "libbeat": "7.4.0", "time": "2019-09-27T06:58:27.000Z", "version": "7.4.0"}}}
2020-06-21T22:05:44.270Z	INFO	[beat]	instance/beat.go:915	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.9"}}}
2020-06-21T22:05:44.270Z	INFO	[beat]	instance/beat.go:919	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-04-08T06:59:14Z","containerized":false,"name":"lts-logging-node-1","ip":["127.0.0.1/8","::1/128","10.3.0.5/24","fe80::20d:3aff:fe20:52a8/64"],"kernel_version":"3.13.0-137-generic","mac":["00:0d:3a:20:52:a8"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"14.04.5 LTS, Trusty Tahr","major":14,"minor":4,"patch":5,"codename":"trusty"},"timezone":"UTC","timezone_offset_sec":0,"id":"ca359a30ee39eda0d5e4dbf9592dd97e"}}}
2020-06-21T22:05:44.271Z	INFO	[beat]	instance/beat.go:948	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/apm-server/bin/apm-server", "name": "apm-server", "pid": 130402, "ppid": 130401, "seccomp": {"mode":"filter"}, "start_time": "2020-06-21T22:05:43.890Z"}}}
2020-06-21T22:05:44.271Z	INFO	instance/beat.go:292	Setup Beat: apm-server; Version: 7.4.0
2020-06-21T22:05:44.271Z	DEBUG	[beat]	instance/beat.go:318	Initializing output plugins
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.271Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.272Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.272Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.272Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.272Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.272Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-06-21T22:05:44.272Z	INFO	elasticsearch/client.go:170	Elasticsearch url: http://10.3.0.8:9200
2020-06-21T22:05:44.272Z	INFO	elasticsearch/client.go:170	Elasticsearch url: http://10.3.0.9:9200
2020-06-21T22:05:44.272Z	INFO	elasticsearch/client.go:170	Elasticsearch url: http://10.3.0.10:9200
2020-06-21T22:05:44.272Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-06-21T22:05:44.272Z	INFO	[publisher]	pipeline/module.go:97	Beat name: lts-logging-node-1
2020-06-21T22:05:44.273Z	INFO	[beater]	beater/beater.go:124	Registering pipeline callback
2020-06-21T22:05:44.273Z	INFO	instance/beat.go:422	apm-server start running.
2020-06-21T22:05:44.273Z	INFO	[beater]	beater/beater.go:259	self instrumentation is disabled
2020-06-21T22:05:44.274Z	INFO	[beater]	beater/beater.go:165	host resolved from localhost:8200 to 127.0.0.1:8200
2020-06-21T22:05:44.274Z	INFO	[handler]	api/mux.go:91	Path / added to request handler
2020-06-21T22:05:44.274Z	INFO	[handler]	api/mux.go:91	Path /assets/v1/sourcemaps added to request handler
2020-06-21T22:05:44.274Z	INFO	[agentcfg]	agentcfg/cache.go:42	Cache creation with default expiration 30s.
2020-06-21T22:05:44.274Z	INFO	[handler]	api/mux.go:91	Path /config/v1/agents added to request handler
2020-06-21T22:05:44.274Z	INFO	[handler]	api/mux.go:91	Path /intake/v2/rum/events added to request handler
2020-06-21T22:05:44.274Z	INFO	[handler]	api/mux.go:91	Path /intake/v2/events added to request handler
2020-06-21T22:05:44.274Z	DEBUG	[kibana]	kibana/connecting_client.go:73	Trying to obtain connection to Kibana.
2020-06-21T22:05:44.274Z	INFO	kibana/client.go:117	Kibana url: http://127.0.0.1:5601
2020-06-21T22:05:44.275Z	INFO	[beater]	beater/server.go:79	Starting apm-server [971d864356e4438bf4a799a1fa052cfd0ce680b4 built 2019-09-27 06:58:27 +0000 UTC]. Hit CTRL-C to stop it.
2020-06-21T22:05:44.275Z	INFO	[beater]	beater/server.go:80	Listening on: 127.0.0.1:8200
2020-06-21T22:05:44.275Z	INFO	[beater]	beater/server.go:91	RUM endpoints disabled.
2020-06-21T22:05:44.275Z	INFO	[beater]	beater/server.go:100	SSL enabled.
2020-06-21T22:05:44.296Z	INFO	[kibana]	kibana/connecting_client.go:80	Successfully obtained connection to Kibana.
2020-06-21T22:05:49.274Z	DEBUG	[publisher]	pipeline/client.go:162	client: closing acker
2020-06-21T22:05:49.274Z	DEBUG	[publisher]	pipeline/client.go:173	client: done closing acker
2020-06-21T22:05:49.274Z	DEBUG	[publisher]	pipeline/client.go:176	client: cancelled 0 events
2020-06-21T22:05:49.274Z	DEBUG	[publisher]	pipeline/client.go:147	client: wait for acker to finish
2020-06-21T22:05:49.274Z	DEBUG	[publisher]	pipeline/client.go:149	client: acker shut down
2020-06-21T22:05:49.274Z	INFO	instance/beat.go:432	apm-server stopped.
2020-06-21T22:05:49.306Z	ERROR	instance/beat.go:878	Exiting: open : no such file or directory

I just looked into this and found that this error message will occur when you have enabled TLS/SSL (ssl.enabled: true), but haven't specified the path to a certificate or key file. If you want APM Server to serve over TLS, then you'll need to provide a certificate and key file.

I've created https://github.com/elastic/apm-server/issues/3908 to improve the error message.

Nice, thanks for your help!