Hi,
File beat version 6.8.13 running on VM, I am trying to parse this ngnix container log into separate fields for reporting on unique clientIP address and browser details to show reports on Kibana.
I added dissect processors in filebeat.yml to token each and every detail in ngnix log which is not showing any changes in my Kibana.
I am seeing "message" attribure or fields with complete ngnix log and is not showing as separate fields, no parsing or tokenizing is happen. Any one have any inputs will be appreciated
Archictecture:
EC2--> filebeat-->kafka topic-->logstash pipline --> Elasticsearch
{"log":"[22/Feb/2021:18:59:51 +0000] nginx:29 10.35.96.46 \"GET /management/health HTTP/1.1\" 200 0 \"-\" \"ELB-HealthChecker/2.0\" \"10.30.70.221\"\n","stream":"stdout","attrs":{"tag":"service=dpeu-dev-dpeu-nginx cluster=dpeu-dev application=dpeu full-imagename=937049000706.dkr.ecr.eu-west-1.amazonaws.com/dpeu/dpeu-nginx:133 container-id=61e6e220e73d full-container-id=61e6e220e73d14b4a9f6c6e8bfa3321c5600817838fe211c5426add29bfda653 log_format=nginx "},"time":"2021-02-22T18:59:51.165741872Z"}
Ngnix container log : just log part
[10/Feb/2021:22:02:32 +0000] nginx:27 10.35.96.46 "GET /sites/default/files/media-icons/default/application-octet-stream.png HTTP/1.1" 200 1520 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.0 Safari/537.36 PTST/1.0" "40.119.52.126, 10.30.69.230, 10.30.70.221"
filebeat.yml
==============
filebeat.inputs:
- enabled: true
fields:
cluster: dpeu-dev
log_topic: filebeat-json
paths:
- /var/lib/docker/containers/*/*-json.log
processors:
- dissect:
when:
contains:
log_format: "nginx"
tokenizer: "[%{@timestamp}] %{*server}:%{&server} %{lbip} \"%{method} %{request} HTTP/%{httpversion}\" %{status} %{bytes} \"%{url}\" \"%{browser}\" \"%{clientip} %{ip1} %{ip2}\""
field: message
target_prefix: "nginx"
seek: cursor
type: log
