Ndjson appears to be ignoring my log line

Elastic Stack Beats
1

Hey,
I am hoping you can point out my silly mistake here.

filebeat.inputs:
  - type: filestream
    id: nginx-stream-json-id
    paths:
      - /var/log/nginx/*.stream.access.log
    parser:
      - ndjson:
          target: "nginx"
          add_error_key: true
output.file:
  path: "/tmp/filebeat"
  filename: filebeat

tail -n1 /var/log/nginx/*.stream.access.log
==> /var/log/nginx/app1.stream.access.log <==
{"source.ip": "127.0.0.1","event.created": "06/Oct/2022:13:40:11 +0100","network.transport": "TCP","http.reponse.status_code": "200","http.response.bytes": "115","http.request.bytes": "2822","event.duration": "0.230","nginx.upstream_addr": "unix:/run/app1/app1-server.sock","nginx.upstream_bytes_sent": "2822","nginx.upstream_bytes_received": "115","nginx.upstream_connect_time": "0.000"}

==> /var/log/nginx/app2.stream.access.log <==
{"source.ip": "127.0.0.1","event.created": "06/Oct/2022:13:42:56 +0100","network.transport": "TCP","http.reponse.status_code": "200","http.response.bytes": "5611","http.request.bytes": "16440","event.duration": "8.050","nginx.upstream_addr": "unix:/run/app2.sock","nginx.upstream_bytes_sent": "16440","nginx.upstream_bytes_received": "5611","nginx.upstream_connect_time": "0.000"}

tail -n1 filebeat-20221006-33.ndjson
{"@timestamp":"2022-10-06T12:43:18.377Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.4.2"},"input":{"type":"filestream"},"host":{"name":"<hostname>"},"agent":{"id":"776f02a7-12e8-4c2b-8ae5-69f9f603d099","name":"<hostname>","type":"filebeat","version":"8.4.2","ephemeral_id":"9ff7cded-536d-4cbd-b1c7-45bf3c96d1a2"},"ecs":{"version":"8.0.0"},"log":{"offset":12930746,"file":{"path":"/var/log/nginx/app2.stream.access.log"}},"message":"{\"source.ip\": \"127.0.0.1\",\"event.created\": \"06/Oct/2022:13:43:16 +0100\",\"network.transport\": \"TCP\",\"http.reponse.status_code\": \"200\",\"http.response.bytes\": \"4371\",\"http.request.bytes\": \"12724\",\"event.duration\": \"4.792\",\"nginx.upstream_addr\": \"unix:/run/app2.sock\",\"nginx.upstream_bytes_sent\": \"12724\",\"nginx.upstream_bytes_received\": \"4371\",\"nginx.upstream_connect_time\": \"0.000\"}"}

Above is a copy of my filebeat.yml configuration, an example of the logs I'm trying to ship and the output of those logs stored in filebeat-20221006-33.ndjson

My issue is that I was expecting ndjson to take the log and parse it for me, however it appears to be ignoring it completely. I am wondering if there is an error in my logs, but I am not getting any feedback

Versions:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-==================================================================
ii  filebeat       8.4.2        amd64        Filebeat sends log files to Logstash or directly to Elasticsearch.
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

Thanks,
Liam

2

Hey @ledzepp4eva, welcome to discuss :slight_smile:

It should be parsers in plural, instead of parser, see the examples in the docs: filestream input | Filebeat Reference [8.4] | Elastic

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.