Can't see journal logs in kibana

Elastic Stack Beats
1

Hello,
I am new to ELK and I'm using elastic cloud on the free trial.

I have setup filebeats on an EC2 machine and 2 log files were available instantly on kibana, which I can filter with fields I have setup on the filebeats.yml for the different files. I have tried doing the same with journalbeats, on the same machine, but after many tries I am unsuccessful.

I have read that logs UI doesn't automatically show the journalbeat indexes, I have tried adding the suggested configuration to my elastic cloud deployment but I get an error because this configuration is not allowed . Since I was unable to have the index be listed, I have tried changing the index on the journalbeat.yml, by adding the following:
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
(under output.elasticsearch) index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
full journalbeat.yml
I have fiddled a bit with template.name/pattern values since I wasn't sure after reading the documentation to what they should be set.

Am I doing something wrong or is this beat just not supported on the elastic cloud yet?

More information:
Ubuntu 18.04.1

filebeat version
filebeat version 6.6.1 (amd64), libbeat 6.6.1 [928f5e3f35fe28c1bd73513ff1cc89406eb212a6 built 2019-02-13 16:12:26 +0000 UTC]
journalbeat version
journalbeat version 6.6.1 (amd64), libbeat 6.6.1 [928f5e3f35fe28c1bd73513ff1cc89406eb212a6 built 2019-02-13 16:17:25 +0000 UTC]
journalctl --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

Also from reading other topics/issues, I have tried looking at the output of lsof | grep beat
but I only see filebeat entries

2

Could you please share the debug logs of Journalbeat? Also, which version are you using?

3

log file, since my last sudo service journalbeat restart this morning: https://gist.github.com/Misirlou/9a7ba73f0b0e18b8ade2534dfc8f8491
assuming you asked for the file /var/log/journalbeat/journalbeat , if it's something else please point me in the right direction

journalbeat version
journalbeat version 6.6.1 (amd64), libbeat 6.6.1 [928f5e3f35fe28c1bd73513ff1cc89406eb212a6 built 2019-02-13 16:17:25 +0000 UTC]

4

I am also able to read the service logs:
journalctl -u consumer_delivery -f
-- Logs begin at Wed 2019-01-30 16:05:54 UTC. --
Feb 25 16:16:30 ip-172-31-0-76 python[26211]: {"evt_name": "DeliveryStockStatusChanged", "event": "firing event...", "logger": "delivery_service.application.signals.handlers.delivery_events_publisher", "level": "info", "timestamp": "2019-02-25T16:16:30.698793Z"}

5

Could you please run journalbeat with ./journalbeat -e -d "*" for ~1-2 mins and paste its output here? Also, what is your systemd version?

6

debug log

systemd --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

journalctl --version gives the same output

Thank you for looking into this :slight_smile:

7

I have changed the
include_matches: [
"systemd.unit=consumer_delivery.service"
]
from just consumer_delivery (I can read the journal with either name) and the output was a lot longer, so part of the problem is this. However it is still not showing up in kibana
debug log

8

What exactly do you mean by not showing up in Kibana? Is it missing from the Discovery page? Is it possible that you are not seeing the events, because the timeframe is not adjusted properly?

9

In the discovery page, I have created the journalbeat-* filter but no data shows up, using last 7 days.
Using the filebeat-* filter, in the fields.app filter there's no delivery-consumer like I configured in journalbeat, but I can see delivery-api which is logged from filebeat in the same machine. Same thing in the infra#/logs page, when I type fields.app: the only suggestion is for delivery-api.

10

I have removed the configs that I was trying to use to change the index to filebeat instead of journalbeat and I can see the logs now in discover under journalbeat.
Thank you for your patience :slight_smile: