# How to parse nginx log using filebeat

**URL:** <https://discuss.elastic.co/t/how-to-parse-nginx-log-using-filebeat/186266>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 18, 2019, 1:42pm UTC](https://discuss.elastic.co/t/how-to-parse-nginx-log-using-filebeat/186266 "2019-06-18T13:42:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![prashantgcloud](https://avatars.discourse-cdn.com/v4/letter/p/a88e4f/32.png) [@prashantgcloud](https://discuss.elastic.co/u/prashantgcloud)\
**Post date:** [June 18, 2019, 1:42pm UTC](https://discuss.elastic.co/t/how-to-parse-nginx-log-using-filebeat/186266/1 "2019-06-18T13:42:14Z")

</div>

I have setup Elasticsearch and kibana using AWS Elastic search service so can't install below plugin :

bin/elasticsearch-plugin install ingest-geoip  
bin/elasticsearch-plugin install ingest-user-agent

I have installed filebeat on EC2 instance using ebextension and it is successfully able to push logs to Elastic search and I'm able to see it on kibana.

Config:

- input\_type: log  
paths:  
- /var/log/nginx/\*.log  
json.message\_key: event  
json.keys\_under\_root: true  
json.overwrite\_keys: true

message:  
xx.xx.xx.xxx - wI485uVG79N7CrcjHx1 [18/Jun/2019:13:17:34 +0000] "POST /v1/cryptoServices/encrypt HTTP/1.1" 200 172 "-" "PostmanRuntime/7.6.0" "644" "0.030" "0.030" "." "prashant" "-" "-" "NO\_ID" "xx.xx.xx.xxx" "-"

However, I want to parse the message with different fields like we can do with nginx module. Is there any other way to achieve this. I don't to setup Logstash on a different server to parse it using grok parser.

Can we achieve it using filebeat on EC2 instance and AWS Elastic search service?

---

<div class="post-metadata">

**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\
**Post date:** [June 19, 2019, 11:53am UTC](https://discuss.elastic.co/t/how-to-parse-nginx-log-using-filebeat/186266/2 "2019-06-19T11:53:41Z")

</div>

Hi @prashantgcloud 🙂

I'm afraid that we don't have knowledge about how the open distro works but in Elasticsearch you can setup an Ingest node with a Grok pattern if you don't use the default formats of Nginx to use the Filebeat Nginx module. This way you can omit using Logstash.

I hope this helps.

---

<div class="post-metadata">

**Author:** ![prashantgcloud](https://avatars.discourse-cdn.com/v4/letter/p/a88e4f/32.png) [@prashantgcloud](https://discuss.elastic.co/u/prashantgcloud)\
**Post date:** [June 25, 2019, 6:29am UTC](https://discuss.elastic.co/t/how-to-parse-nginx-log-using-filebeat/186266/4 "2019-06-25T06:29:11Z")

</div>

I can use filebeat nginx module, but then I can't install below plugin on Elastic search instance as I have set it up using AWS Elasticsearch service.

bin/elasticsearch-plugin install ingest-geoip  
bin/elasticsearch-plugin install ingest-user-agent..

My question is how can I send parsed nginx log instead of sending it in a single message:

Current logs:  
message: xx.xx.xx.xxx - wI485uVG79N7CrcjHx1 [18/Jun/2019:13:17:34 +0000] "POST /v1/cryptoServices/encrypt HTTP/1.1" 200 172 "-" "PostmanRuntime/7.6.0" "644" "0.030" "0.030" "." "prashant" "-" "-" "NO\_ID" "xx.xx.xx.xxx" "-"

Expected logs:  
nginx.access.reponse\_code  
nginx.access.user\_agent and so on

So I can create better dashboard.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 23, 2019, 6:29am UTC](https://discuss.elastic.co/t/how-to-parse-nginx-log-using-filebeat/186266/5 "2019-07-23T06:29:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.