Hey,
I am hoping you can point out my silly mistake here.
filebeat.inputs:
- type: filestream
id: nginx-stream-json-id
paths:
- /var/log/nginx/*.stream.access.log
parser:
- ndjson:
target: "nginx"
add_error_key: true
output.file:
path: "/tmp/filebeat"
filename: filebeat
tail -n1 /var/log/nginx/*.stream.access.log
==> /var/log/nginx/app1.stream.access.log <==
{"source.ip": "127.0.0.1","event.created": "06/Oct/2022:13:40:11 +0100","network.transport": "TCP","http.reponse.status_code": "200","http.response.bytes": "115","http.request.bytes": "2822","event.duration": "0.230","nginx.upstream_addr": "unix:/run/app1/app1-server.sock","nginx.upstream_bytes_sent": "2822","nginx.upstream_bytes_received": "115","nginx.upstream_connect_time": "0.000"}
==> /var/log/nginx/app2.stream.access.log <==
{"source.ip": "127.0.0.1","event.created": "06/Oct/2022:13:42:56 +0100","network.transport": "TCP","http.reponse.status_code": "200","http.response.bytes": "5611","http.request.bytes": "16440","event.duration": "8.050","nginx.upstream_addr": "unix:/run/app2.sock","nginx.upstream_bytes_sent": "16440","nginx.upstream_bytes_received": "5611","nginx.upstream_connect_time": "0.000"}
tail -n1 filebeat-20221006-33.ndjson
{"@timestamp":"2022-10-06T12:43:18.377Z","@metadata":{"beat":"filebeat","type":"_doc","version":"8.4.2"},"input":{"type":"filestream"},"host":{"name":"<hostname>"},"agent":{"id":"776f02a7-12e8-4c2b-8ae5-69f9f603d099","name":"<hostname>","type":"filebeat","version":"8.4.2","ephemeral_id":"9ff7cded-536d-4cbd-b1c7-45bf3c96d1a2"},"ecs":{"version":"8.0.0"},"log":{"offset":12930746,"file":{"path":"/var/log/nginx/app2.stream.access.log"}},"message":"{\"source.ip\": \"127.0.0.1\",\"event.created\": \"06/Oct/2022:13:43:16 +0100\",\"network.transport\": \"TCP\",\"http.reponse.status_code\": \"200\",\"http.response.bytes\": \"4371\",\"http.request.bytes\": \"12724\",\"event.duration\": \"4.792\",\"nginx.upstream_addr\": \"unix:/run/app2.sock\",\"nginx.upstream_bytes_sent\": \"12724\",\"nginx.upstream_bytes_received\": \"4371\",\"nginx.upstream_connect_time\": \"0.000\"}"}
Above is a copy of my filebeat.yml configuration, an example of the logs I'm trying to ship and the output of those logs stored in filebeat-20221006-33.ndjson
My issue is that I was expecting ndjson to take the log and parse it for me, however it appears to be ignoring it completely. I am wondering if there is an error in my logs, but I am not getting any feedback
Versions:
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-==================================================================
ii filebeat 8.4.2 amd64 Filebeat sends log files to Logstash or directly to Elasticsearch.
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy
Thanks,
Liam
