JSON logs not being uploaded

[etoews (Everett)]

I've configured Filebeat to harvest some OpenShift Audit logs. The logs are in JSON format so I'm using the JSON Log input.

However, the logs simply aren't getting uploaded to ElasticSearch (via Logstash). Well...I'm unable to see them in Kibana when I search for "auditID", which is one of the fields as you can see from the Log output below. I've also included as much debugging info as I can think of below.

Can anyone spot any misconfiguration or anything else I might be missing?

Any other debugging tips for this?

Config

# grep -A 9 filebeat.inputs /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: docker
  containers.ids: '*'
  processors:
  - add_docker_metadata: ~
  - add_host_metadata: ~
- type: log
  paths:
    - /var/log/audit/openshift.log
  json.keys_under_root: true

Log output

# tail -f /var/log/audit/openshift.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-02T05:27:49Z"},"level":"Metadata","timestamp":"2018-10-02T05:27:49Z","auditID":"d65a7719-d789-4dbb-9a36-0944113259b3","stage":"RequestReceived","requestURI":"xxx","verb":"get","user":{},"sourceIPs":["x.x.x.x"],"requestReceivedTimestamp":"2018-10-02T05:27:49.192712Z","stageTimestamp":"2018-10-02T05:27:49.192712Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2018-10-02T05:27:49Z"},"level":"Metadata","timestamp":"2018-10-02T05:27:49Z","auditID":"d65a7719-d789-4dbb-9a36-0944113259b3","stage":"ResponseComplete","requestURI":"xxx","verb":"get","user":{},"sourceIPs":["x.x.x.x"],"responseStatus":{"metadata":{},"code":302},"requestReceivedTimestamp":"2018-10-02T05:27:49.192712Z","stageTimestamp":"2018-10-02T05:27:49.194596Z"}
...

Version

# filebeat version
filebeat version 6.4.1 (amd64), libbeat 6.4.1 [37b5f2d2a20f2734b2373a454b4b4cbb2627e841 built 2018-09-13 21:25:47 +0000 UTC]

Debug Mode

# filebeat -c /etc/filebeat/filebeat.yml  -e -d *
2018-10-02T18:27:07.604+1300    INFO    instance/beat.go:544    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-10-02T18:27:07.604+1300    INFO    instance/beat.go:551    Beat UUID: 2cafaf0a-c529-4f84-822f-67c74e45e052
2018-10-02T18:27:07.605+1300    INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
2018-10-02T18:27:07.605+1300    INFO    [beat]  instance/beat.go:768    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "2cafaf0a-c529-4f84-822f-67c74e45e052"}}}
2018-10-02T18:27:07.605+1300    INFO    [beat]  instance/beat.go:777    Build info      {"system_info": {"build": {"commit": "37b5f2d2a20f2734b2373a454b4b4cbb2627e841", "libbeat": "6.4.1", "time": "2018-09-13T21:25:47.000Z", "version": "6.4.1"}}}
2018-10-02T18:27:07.605+1300    INFO    [beat]  instance/beat.go:780    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
...
2018-10-02T18:27:07.643+1300    INFO    log/input.go:138        Configured paths: [/var/log/audit/openshift.log]
...

Connectivity

# filebeat test output
logstash: foo1.com:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.x
    dial up... OK
  TLS... OK
  talk to server... OK
logstash: foo2.com:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.x
    dial up... OK
  TLS... OK
  talk to server... OK

[kvch (Noémi Ványi)]

For testing purposes could you please set a different registry file for Filebeat and try to send the logs again with debug logging turned on. Please share the ouput of at least the first 20-30 seconds, so we can get an idea of what is happening in your instance.

You can set the registry file to a different path:

filebeat.registry: ${path.data}/test-registry

By setting it to a different path we can try to send events without losing where you left off. You can also delete the registry file, so Filebeat will reread everything it has encountered before. But for a first debug run it's not necessary.
First you could try sending it to a file input, to see if Filebeat is able to find and process your logs.

[etoews (Everett)]

Sorry, I'm not very familiar with Beats at all. I don't really know what "try sending it to a file input" means. How would I do that?

Thanks!

[kvch (Noémi Ványi)]

Apparently, I am not familiar with the English language... I am sorry I meant file output.
Just add the following snippet to your config and comment out the existing output config.

output.file:
  path: "/path/to/the/input/filebeat"
  filename: filebeat

See more: https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html