How to parse nginx log using filebeat

[prashantgcloud (Prashant Goel)]

I have setup Elasticsearch and kibana using AWS Elastic search service so can't install below plugin :

bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent

I have installed filebeat on EC2 instance using ebextension and it is successfully able to push logs to Elastic search and I'm able to see it on kibana.

Config:

• input_type: log
  paths:
  - /var/log/nginx/*.log
  json.message_key: event
  json.keys_under_root: true
  json.overwrite_keys: true

message:
xx.xx.xx.xxx - wI485uVG79N7CrcjHx1 [18/Jun/2019:13:17:34 +0000] "POST /v1/cryptoServices/encrypt HTTP/1.1" 200 172 "-" "PostmanRuntime/7.6.0" "644" "0.030" "0.030" "." "prashant" "-" "-" "NO_ID" "xx.xx.xx.xxx" "-"

However, I want to parse the message with different fields like we can do with nginx module. Is there any other way to achieve this. I don't to setup Logstash on a different server to parse it using grok parser.

Can we achieve it using filebeat on EC2 instance and AWS Elastic search service?

[Mario_Castro]

Hi @prashantgcloud

I'm afraid that we don't have knowledge about how the open distro works but in Elasticsearch you can setup an Ingest node with a Grok pattern if you don't use the default formats of Nginx to use the Filebeat Nginx module. This way you can omit using Logstash.

I hope this helps.

[prashantgcloud (Prashant Goel)]

I can use filebeat nginx module, but then I can't install below plugin on Elastic search instance as I have set it up using AWS Elasticsearch service.

bin/elasticsearch-plugin install ingest-geoip
bin/elasticsearch-plugin install ingest-user-agent..

My question is how can I send parsed nginx log instead of sending it in a single message:

Current logs:
message: xx.xx.xx.xxx - wI485uVG79N7CrcjHx1 [18/Jun/2019:13:17:34 +0000] "POST /v1/cryptoServices/encrypt HTTP/1.1" 200 172 "-" "PostmanRuntime/7.6.0" "644" "0.030" "0.030" "." "prashant" "-" "-" "NO_ID" "xx.xx.xx.xxx" "-"

Expected logs:
nginx.access.reponse_code
nginx.access.user_agent and so on

So I can create better dashboard.