Nginx file_format incorrect output

Hello,

I have a question about the nginx log_format. Currently we have the problem, that the source.address in filebeat shows address and ip in one field.

For example:

"source": {
  "address": "www.domain.tld 123.123.123.123"
},

This is the config of our nginx file_format (nginx.conf):

    log_format filebeat '$http_host $remote_addr - $remote_user [$time_local] '
                   '"$request" $status $bytes_sent '
                   '"$http_referer" "$http_user_agent" "$request_time"';

We are using the newest nginx filebeat module (master branch) from https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json

We want to split the ip into the field source.ip. Maybe someone can help me?

Thank you!

Regards!

Hi, This should do the trick

 mutate {
                split => ["address"," "]
                add_field => { "address" => "%{[address][0]}" }
                add_field => { "ip" => "%{[address][1]}" }
}

let me know

Thank you very much! But now I'm stumped where I have to add this function into the default.json (https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json).

Hey, are you processing data through elasticsearch with logstash ?

Hey,

yes, that's correct. We're using filebeat on the web node as service with the filebeat nginx module. And in our nginx.conf we're using the file_format as you can see above.

So if you use logstash you can actually change this using the filter plugin and the mutate in your logstash pipeline

Let me know if you need some more help.

Hello.

Unfortunately the issue is still persist.

As you can see, the source.address shows domain and ip. But the source.ip field is empty. But why? I can't add your example in my .json file, because there is no logstash service. We are only using filebeat.

My ingest .json file:

{
"description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
"processors": [
    {
        "grok": {
            "field": "message",
            "patterns": [
                "%{NGINX_HOST} - %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\""
            ],
            "pattern_definitions": {
                "NGINX_HOST": "(?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR:destination.domain})(:%{NUMBER:destination.port})?",
                "NGINX_NOTSEPARATOR": "[^\t ,:]+",
                "NGINX_ADDRESS_LIST": "(?:%{IP}|%{WORD})(\"?,?\\s*(?:%{IP}|%{WORD}))*"
            },
            "ignore_missing": true
        }
    },
    {
        "set": {
            "field": "error.message",
            "value": "{{ _ingest.on_failure_message }}"
        }
    }
]
}

Thank you.

Regards

Use grok,

In your ingest file create another grok after the first one.

I can think of something like :

source field : source.address

(?(\w+\S+))%{SPACE}(?%{IPV4})

This will split the field using space and create 2 fields

  • DomainName
  • IP

Thank you. I have add another grok like this:

        {
        "grok": {
            "field": "source.address",
            "patterns": [
                "(?(\w+\S+))%{SPACE}(?%{IP:source.ip})"
            ],
            "ignore_missing": true
        }
    },

I think there is some issue, but I can't find the error. The logging isn't working anymore.