Nginx file_format incorrect output

[marbro15]

Hello,

I have a question about the nginx log_format. Currently we have the problem, that the source.address in filebeat shows address and ip in one field.

For example:

"source": {
  "address": "www.domain.tld 123.123.123.123"
},

This is the config of our nginx file_format (nginx.conf):

    log_format filebeat '$http_host $remote_addr - $remote_user [$time_local] '
                   '"$request" $status $bytes_sent '
                   '"$http_referer" "$http_user_agent" "$request_time"';

We are using the newest nginx filebeat module (master branch) from https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json

We want to split the ip into the field source.ip. Maybe someone can help me?

Thank you!

Regards!

[grumo35]

Hi, This should do the trick

 mutate {
                split => ["address"," "]
                add_field => { "address" => "%{[address][0]}" }
                add_field => { "ip" => "%{[address][1]}" }
}

let me know

[marbro15]

Thank you very much! But now I'm stumped where I have to add this function into the default.json (https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/ingest/default.json).

[grumo35]

Hey, are you processing data through elasticsearch with logstash ?

[marbro15]

Hey,

yes, that's correct. We're using filebeat on the web node as service with the filebeat nginx module. And in our nginx.conf we're using the file_format as you can see above.

[grumo35]

So if you use logstash you can actually change this using the filter plugin and the mutate in your logstash pipeline

Let me know if you need some more help.

[marbro15]

Hello.

Unfortunately the issue is still persist.

As you can see, the source.address shows domain and ip. But the source.ip field is empty. But why? I can't add your example in my .json file, because there is no logstash service. We are only using filebeat.

My ingest .json file:

{
"description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
"processors": [
    {
        "grok": {
            "field": "message",
            "patterns": [
                "%{NGINX_HOST} - %{DATA:user.name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{DATA:nginx.access.info}\" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"%{DATA:http.request.referrer}\" \"%{DATA:user_agent.original}\""
            ],
            "pattern_definitions": {
                "NGINX_HOST": "(?:%{IP:destination.ip}|%{NGINX_NOTSEPARATOR:destination.domain})(:%{NUMBER:destination.port})?",
                "NGINX_NOTSEPARATOR": "[^\t ,:]+",
                "NGINX_ADDRESS_LIST": "(?:%{IP}|%{WORD})(\"?,?\\s*(?:%{IP}|%{WORD}))*"
            },
            "ignore_missing": true
        }
    },
    {
        "set": {
            "field": "error.message",
            "value": "{{ _ingest.on_failure_message }}"
        }
    }
]
}

Thank you.

Regards

[grumo35]

Use grok,

In your ingest file create another grok after the first one.

I can think of something like :

source field : source.address

(?(\w+\S+))%{SPACE}(?%{IPV4})

This will split the field using space and create 2 fields

• DomainName
• IP

[marbro15]

Thank you. I have add another grok like this:

        {
        "grok": {
            "field": "source.address",
            "patterns": [
                "(?(\w+\S+))%{SPACE}(?%{IP:source.ip})"
            ],
            "ignore_missing": true
        }
    },

I think there is some issue, but I can't find the error. The logging isn't working anymore.