Adding more fields from the nginx log file

[gabriel_tessier]

Hi,

I'm using Filebeat with Kibana and Elastic 5.5.0 to get the nginx data from logs file and it's work well, but all the data from my nginx log file was not in Kibana.

Here an example of my log file:

10.10.10.10 - - [17/Oct/2017:03:48:00 +0200] "GET /myurl/40 HTTP/1.1" 200 75793 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "0.277"

the last "0.277" were not in Kibana maybe because it's not following the standard format pattern.

According on what I red here an there the file format is defined in the following pipline:
GET _ingest/pipeline/filebeat-5.6.2-nginx-access-default

So I tried to update but didn't see any change:

"patterns": [
            """"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} "%{DATA:nginx.access.referrer}" "%{DATA:nginx.access.agent}" "%{NUMBER:nginx.request_time}" """
          ]

I added the "%{NUMBER:nginx.request_time}" at the end

According to the documentation about the pipeline:

Create or update pipeline API | Elasticsearch Guide [8.11] | Elastic

...pipeline changes take effect immediately.

So I expect to have nothing else to do??!!

Not related but when I edited the pipline to update it. Kibana return a syntax error, and I need to add a space before the last 3 """, if it can help somebody.

I also refresh the fields in Kibana but didn't see any new field.

Thanks for any help!

[gabriel_tessier]

Got it!!

There's was a problem in the definition of the patterns from the nginx conf file:

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$request_time"';

The last $request_time if you surround with " you need to put a space, that's why in Kibana it was complaining.

I solve it by removing the " around the $request_time in the xginx conf file, but if you need to put " you need to put a space after.

All this can be tested with this command:

POST _ingest/pipeline/filebeat-5.6.2-nginx-access-default/_simulate
{
"docs":[
  {
    "_source": {
      "message": "10.10.10.10 - - [17/Oct/2017:03:48:00 +0200] \"GET /my_page/40 HTTP/1.1\" 200 75793 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" 0.277"
    }
  }
  ]
}

But need to fix the patterns first by removing the " or adding space.

Thank you Elastic guys and girls to make so easy to use and good product!
Sorry for the noise but hope that it can help somebody else.