# Parsing NGINX for Filebeat dashboards with Logstash **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [February 12, 2020, 8:30am UTC](https://discuss.elastic.co/t/parsing-nginx-for-filebeat-dashboards-with-logstash/218924 "2020-02-12T08:30:31Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![MerceneX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mercenex/32/62509_2.png) [@MerceneX](https://discuss.elastic.co/u/MerceneX)\ **Post date:** [February 12, 2020, 8:30am UTC](https://discuss.elastic.co/t/parsing-nginx-for-filebeat-dashboards-with-logstash/218924/1 "2020-02-12T08:30:31Z")
Hi! I've been using Logstash to parse custom logs with great success. What I am having trouble with now, is using that same Logstash and Filebeat configuration to get NGINX logs across. I can get logs to my Kibana instance, the problem is that the fields are named differently to what the Filebeat dashboards are requesting. Metricbeat worked like a charm, nothing was necessary to setup in the Logstash pipeline and it was parsed and plugged without a problem to the dashboards. Filebeat on the other hand is a pain. I'm sending over Syslogs and NGINX, none are getting parsed if I set nothing for them in the Logstash pipeline, but if I setup the pipeline specified here for these modules : [https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html#parsing-nginx](https://www.elastic.co/guide/en/logstash/current/logstash-config-for-filebeat-modules.html#parsing-nginx) they get parsed, but with different names. Please help, I've been at this for two days now to no avail. If you need any more information I am happy to provide it. This is the Logstash pipeline: input { beats { port =\> 5044 } stdin { } } filter { if [event][dataset] == "nginx.access" { } else if [event][name] == "nginx.error" { } else if [event][dataset] == "apache.access" { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}"} } mutate { add\_field =\> { "[@metadata][service\_type]" =\> "apache-access" } add\_field =\> { "[@metadata][provider\_identifier]" =\> "%{[clientip]}" } } } else if [event][module] == "system" { mutate { add\_field =\> { "[@metadata][service\_type]" =\> "system" } add\_field =\> { "[@metadata][provider\_identifier]" =\> "%{[agent][hostname]}" } } } else if[fields][log\_type] == "appLog" { json { source =\> "message" } mutate { add\_field =\> { "[@metadata][service\_type]" =\> "appLog" } add\_field =\> { "[@metadata][provider\_identifier]" =\> "%{[agent][hostname]}" } replace =\> { "[@metadata][beat]" =\> "applog-filebeat" } } } if "\_grokparsefailure" in [tags] { mutate { replace =\> { "[@metadata][provider\_identifier]" =\> "grok\_parse\_failure" } } } if ![@metadata][service\_type] { mutate { add\_field =\> { "[@metadata][service\_type]" =\> "generic" } } } if ![@metadata][provider\_identifier] { mutate { add\_field =\> { "[@metadata][provider\_identifier]" =\> "%{[agent.hostname]}" } } } mutate { lowercase =\> ["[@metadata][provider\_identifier]" ] lowercase =\> ["[@metadata][service\_type]" ] } } output { elasticsearch { hosts =\> "elasticsearch:9200" user =\> "elastic" password =\> "admin" index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-agent-for-%{[@metadata][service\_type]}-%{[@metadata][provider\_identifier]}" } stdout { codec =\> rubydebug { metadata =\> true } } } And the Filebeat.yml file: filebeat.inputs: - type: log enabled: true paths: - /var/log/banka/_\_general.json fields: log\_type: appLog app: SecureBank app\_log\_type: general - type: log enabled: false paths: - /var/log/banka/_\_global\_exception.json fields: log\_type: appLog app: SecureBank app\_log\_type: global\_exception - type: log enabled: true paths: - /var/log/banka/_\_revision.json fields: log\_type: appLog app: SecureBank app\_log\_type: revision filebeat.config.modules: path: ${path.config}/modules.d/_.yml reload.enabled: false setup.template.settings: index.number\_of\_shards: 1 output.logstash: hosts: ["localhost:5044"] #output.elasticsearch: # hosts: ["localhost:9200"] #username: "elastic" #password: "admin" #setup.kibana: # host: "localhost:5601" #username: "elastic" # password: "admin" processors: - add\_host\_metadata: ~ - add\_cloud\_metadata: ~ Thank you to anyone, who took the time reading this. ---
**Author:** ![MerceneX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mercenex/32/62509_2.png) [@MerceneX](https://discuss.elastic.co/u/MerceneX)\ **Post date:** [February 12, 2020, 12:37pm UTC](https://discuss.elastic.co/t/parsing-nginx-for-filebeat-dashboards-with-logstash/218924/2 "2020-02-12T12:37:44Z")
Well I managed to figure it out. In my case I'm using elasticsearch ingest pipelines for parsing the nginx module. All I needed was to setup the Logstash pipeline output accordingly: output { if [@metadata][pipeline] { elasticsearch { hosts =\> "elasticsearch:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" pipeline =\> "%{[@metadata][pipeline]}" user =\> "elastic" password =\> "admin" } } else { elasticsearch { hosts =\> "elasticsearch:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" user =\> "elastic" password =\> "admin" } } stdout { codec =\> rubydebug { metadata =\> true } } } Hope this helps anyone. You can read more in the documentation: [https://www.elastic.co/guide/en/logstash/7.5/use-ingest-pipelines.html](https://www.elastic.co/guide/en/logstash/7.5/use-ingest-pipelines.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 11, 2020, 12:37pm UTC](https://discuss.elastic.co/t/parsing-nginx-for-filebeat-dashboards-with-logstash/218924/3 "2020-03-11T12:37:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.