Journalbeat exits with success

chrissound · June 13, 2020, 4:48pm

[root@XenonKiloCranberry:~]# /nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin/journalbeat -c /nix/store/9g135rj3vz502l03iy08567nyl22iirm-journalbeat.yml -path.data /var/lib/journalbeat/data -path.logs /var/lib/journalbeat/logs

[root@XenonKiloCranberry:~]# echo $? 
0

The config file:

[root@XenonKiloCranberry:~]# cat /nix/store/9g135rj3vz502l03iy08567nyl22iirm-journalbeat.yml
name: journalbeat
tags: []



journalbeat:
  seek_position: cursor
  cursor_seek_fallback: tail
  write_cursor_state: true
  cursor_flush_period: 5s
  clean_field_names: true
  convert_to_numbers: false
  move_metadata_to_field: journal
  default_type: journal
output.elasticsearch:
  hosts: ["localhost:9200"]

My expectation is this program would run as continuously running daemon, and if not, it would at least output an error if something goes wrong.

Version:

[root@XenonKiloCranberry:~]# /nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin/journalbeat version
journalbeat version 6.8.3 (amd64), libbeat 6.8.3 [unknown built unknown]

jsoriano · June 15, 2020, 4:16pm

Hi @chrissound, welcome to discuss

Yes, journalbeat should actually run as a daemon, and write something if fails to start.

Maybe you can find something in the logs path specified in the command line?

You can also use the -e flag, that overwrites any other logging configuration and makes journalbeat to log to stderr.

chrissound · June 15, 2020, 5:20pm

Thanks. It seems to not run for very long in that case, (with the addition of the -e following:

/nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin/journalbeat -e -c /nix/store/9g135rj3vz502l03iy08567nyl22iirm-journalbeat.yml -path.data /var/lib/journalbeat/data -path.logs /var/lib/journalbeat/logs
    2020-06-15T18:17:54.192+0100	INFO	instance/beat.go:611	Home path: [/nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin] Config path: [/nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin] Data path: [/var/lib/journalbeat/data] Logs path: [/var/lib/journalbeat/logs]
    2020-06-15T18:17:54.192+0100	INFO	instance/beat.go:618	Beat UUID: 5fd1bd17-1bf6-4c14-abb9-7447f92ec5e0
    2020-06-15T18:17:54.192+0100	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
    2020-06-15T18:17:54.192+0100	INFO	[beat]	instance/beat.go:931	Beat info	{"system_info": {"beat": {"path": {"config": "/nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin", "data": "/var/lib/journalbeat/data", "home": "/nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin", "logs": "/var/lib/journalbeat/logs"}, "type": "journalbeat", "uuid": "5fd1bd17-1bf6-4c14-abb9-7447f92ec5e0"}}}
    2020-06-15T18:17:54.192+0100	INFO	[beat]	instance/beat.go:940	Build info	{"system_info": {"build": {"commit": "unknown", "libbeat": "6.8.3", "time": "1754-08-30T22:43:41.128Z", "version": "6.8.3"}}}
    2020-06-15T18:17:54.192+0100	INFO	[beat]	instance/beat.go:943	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.1"}}}
    2020-06-15T18:17:54.195+0100	INFO	[beat]	instance/beat.go:947	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-06-12T21:23:07+01:00","containerized":false,"name":"XenonKiloCranberry","ip":["127.0.0.1/8","::1/128","94.130.178.87/32","fe80::9400:ff:fe43:4fd/64","172.17.0.1/16","fe80::d017:8ff:fea7:a3d8/64","10.100.0.1/24","fe80::5853:84ff:fecc:a997/64","fe80::e857:82ff:fecf:136f/64"],"kernel_version":"4.19.105","mac":["96:00:00:43:04:fd","d2:17:08:a7:a3:d8","5a:53:84:cc:a9:97","ea:57:82:cf:13:6f"],"os":{"family":"","platform":"nixos","name":"NixOS","version":"20.03.2133.8946799e079 (Markhor)","major":20,"minor":3,"patch":2133,"codename":"markhor"},"timezone":"BST","timezone_offset_sec":3600,"id":"3b69b17f9bc74de2900b86176785b839"}}}
    2020-06-15T18:17:54.195+0100	INFO	[beat]	instance/beat.go:976	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/nix/store/lfcgbj5s5iwd4fa6dcs2lmrkryy03a9s-journalbeat-6.8.3-bin/bin/journalbeat", "name": "journalbeat", "pid": 25461, "ppid": 25344, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-06-15T18:17:53.300+0100"}}}
    2020-06-15T18:17:54.195+0100	INFO	instance/beat.go:280	Setup Beat: journalbeat; Version: 6.8.3
    2020-06-15T18:17:54.195+0100	INFO	elasticsearch/client.go:164	Elasticsearch url: http://localhost:9200
    2020-06-15T18:17:54.196+0100	INFO	[publisher]	pipeline/module.go:110	Beat name: journalbeat
    2020-06-15T18:17:54.196+0100	WARN	[cfgwarn]	beater/journalbeat.go:50	EXPERIMENTAL: Journalbeat is experimental.
    2020-06-15T18:17:54.197+0100	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
    2020-06-15T18:17:54.198+0100	INFO	instance/beat.go:402	journalbeat start running.
    2020-06-15T18:17:54.198+0100	INFO	[journalbeat]	beater/journalbeat.go:104	journalbeat is running! Hit CTRL-C to stop it.
    2020-06-15T18:17:54.198+0100	INFO	[journalbeat]	beater/journalbeat.go:129	journalbeat is stopping
    2020-06-15T18:17:54.200+0100	INFO	[monitoring]	log/log.go:152	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":6}},"total":{"ticks":10,"time":{"ms":19},"value":10},"user":{"ticks":10,"time":{"ms":13}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":7},"info":{"ephemeral_id":"e8ce02dd-d768-407a-a442-5317125608a0","uptime":{"ms":11}},"memstats":{"gc_next":4473924,"memory_alloc":3112488,"memory_total":3112488,"rss":21897216}},"journalbeat":{"libbeat":{"output":{"type":"elasticsearch"}},"system":{"cpu":{"cores":2},"load":{"1":0.04,"5":0.02,"norm":{"1":0.02,"5":0.01}}}}}}}
    2020-06-15T18:17:54.200+0100	INFO	[monitoring]	log/log.go:153	Uptime: 14.145873ms
    2020-06-15T18:17:54.200+0100	INFO	[monitoring]	log/log.go:130	Stopping metrics logging.
    2020-06-15T18:17:54.200+0100	INFO	instance/beat.go:412	journalbeat stopped.

jsoriano · June 15, 2020, 6:10pm

Umm, taking a look to the configuration I wonder if we are talking about the same journalbeat

There is this Journalbeat: https://github.com/mheese/journalbeat, that is superseeded by the official journalbeat: https://www.elastic.co/guide/en/beats/journalbeat/7.7/index.html.

I would recommend you to use the second one.

chrissound · June 15, 2020, 6:54pm

I seem to get a similar issue with journalbeat 7.5 (seems to be the latest supported by the distro).

[root@XenonKiloCranberry:/var/lib/journalbeat/logs]# cat journalbeat
2020-06-15T19:48:38.470+0100	INFO	instance/beat.go:610	Home path: [/nix/store/j8qdszj53fmwhykk41wpiyb1fpap1g8p-journalbeat-7.5.1-bin/bin] Config path: [/nix/store/j8qdszj53fmwhykk41wpiyb1fpap1g8p-journalbeat-7.5.1-bin/bin] Data path: [/var/lib/journalbeat/data] Logs path: [/var/lib/journalbeat/logs]
2020-06-15T19:48:38.471+0100	INFO	instance/beat.go:618	Beat ID: 5fd1bd17-1bf6-4c14-abb9-7447f92ec5e0
2020-06-15T19:48:38.471+0100	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-06-15T19:48:38.471+0100	INFO	[beat]	instance/beat.go:941	Beat info	{"system_info": {"beat": {"path": {"config": "/nix/store/j8qdszj53fmwhykk41wpiyb1fpap1g8p-journalbeat-7.5.1-bin/bin", "data": "/var/lib/journalbeat/data", "home": "/nix/store/j8qdszj53fmwhykk41wpiyb1fpap1g8p-journalbeat-7.5.1-bin/bin", "logs": "/var/lib/journalbeat/logs"}, "type": "journalbeat", "uuid": "5fd1bd17-1bf6-4c14-abb9-7447f92ec5e0"}}}
2020-06-15T19:48:38.471+0100	INFO	[beat]	instance/beat.go:950	Build info	{"system_info": {"build": {"commit": "unknown", "libbeat": "7.5.1", "time": "1754-08-30T22:43:41.128Z", "version": "7.5.1"}}}
2020-06-15T19:48:38.471+0100	INFO	[beat]	instance/beat.go:953	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.1"}}}
2020-06-15T19:48:38.472+0100	INFO	[beat]	instance/beat.go:957	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-06-12T21:23:07+01:00","containerized":false,"name":"XenonKiloCranberry","ip":["127.0.0.1/8","::1/128","94.130.178.87/32","fe80::9400:ff:fe43:4fd/64","172.17.0.1/16","fe80::d017:8ff:fea7:a3d8/64","10.100.0.1/24","fe80::5853:84ff:fecc:a997/64","fe80::e857:82ff:fecf:136f/64"],"kernel_version":"4.19.105","mac":["96:00:00:43:04:fd","d2:17:08:a7:a3:d8","5a:53:84:cc:a9:97","ea:57:82:cf:13:6f"],"os":{"family":"","platform":"nixos","name":"NixOS","version":"20.03.2133.8946799e079 (Markhor)","major":20,"minor":3,"patch":2133,"codename":"markhor"},"timezone":"BST","timezone_offset_sec":3600,"id":"3b69b17f9bc74de2900b86176785b839"}}}
2020-06-15T19:48:38.473+0100	INFO	[beat]	instance/beat.go:986	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/", "exe": "/nix/store/j8qdszj53fmwhykk41wpiyb1fpap1g8p-journalbeat-7.5.1-bin/bin/journalbeat", "name": "journalbeat", "pid": 24434, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-06-15T19:48:37.560+0100"}}}
2020-06-15T19:48:38.473+0100	INFO	instance/beat.go:297	Setup Beat: journalbeat; Version: 7.5.1
2020-06-15T19:48:38.473+0100	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'journalbeat-7.5.1' as ILM is enabled.
2020-06-15T19:48:38.473+0100	INFO	elasticsearch/client.go:171	Elasticsearch url: http://localhost:9200
2020-06-15T19:48:38.474+0100	INFO	[publisher]	pipeline/module.go:97	Beat name: journalbeat
2020-06-15T19:48:38.474+0100	WARN	[cfgwarn]	beater/journalbeat.go:53	EXPERIMENTAL: Journalbeat is experimental.
2020-06-15T19:48:38.477+0100	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-06-15T19:48:38.477+0100	INFO	instance/beat.go:429	journalbeat start running.
2020-06-15T19:48:38.477+0100	INFO	[journalbeat]	beater/journalbeat.go:91	journalbeat is running! Hit CTRL-C to stop it.
2020-06-15T19:48:38.477+0100	INFO	[journalbeat]	beater/journalbeat.go:116	journalbeat is stopping
2020-06-15T19:48:38.479+0100	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":13}},"total":{"ticks":30,"time":{"ms":43},"value":30},"user":{"ticks":20,"time":{"ms":30}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"b29315a5-908c-4c3e-82e8-bf44acc59336","uptime":{"ms":25}},"memstats":{"gc_next":4958960,"memory_alloc":2968024,"memory_total":5332384,"rss":33562624},"runtime":{"goroutines":12}},"journalbeat":{"libbeat":{"output":{"type":"elasticsearch"}},"system":{"cpu":{"cores":2},"load":{"1":1.42,"15":0.81,"5":1.74,"norm":{"1":0.71,"15":0.405,"5":0.87}}}}}}}
2020-06-15T19:48:38.479+0100	INFO	[monitoring]	log/log.go:154	Uptime: 27.145317ms
2020-06-15T19:48:38.479+0100	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2020-06-15T19:48:38.479+0100	INFO	instance/beat.go:435	journalbeat stopped.

chrissound · June 15, 2020, 6:58pm

I've decided to not go further with this. Sorry for any inconvenience.

jsoriano · June 16, 2020, 10:11am

Thanks for the update.

Just for the record if someone has the same issue, if using the official journalbeat, the issue maybe that the configuration doesn't contain any input: https://www.elastic.co/guide/en/beats/journalbeat/7.7/journalbeat-configuration.html

In any case journalbeat should probably complain if that is the case.

system · July 14, 2020, 12:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.