# Adding more fields from the nginx log file

**URL:** <https://discuss.elastic.co/t/adding-more-fields-from-the-nginx-log-file/104231>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 17, 2017, 11:14am UTC](https://discuss.elastic.co/t/adding-more-fields-from-the-nginx-log-file/104231 "2017-10-17T11:14:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gabriel\_tessier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabriel_tessier/32/27911_2.png) [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)\
**Post date:** [October 17, 2017, 11:14am UTC](https://discuss.elastic.co/t/adding-more-fields-from-the-nginx-log-file/104231/1 "2017-10-17T11:14:35Z")

</div>

Hi,

I'm using Filebeat with Kibana and Elastic 5.5.0 to get the nginx data from logs file and it's work well, but all the data from my nginx log file was not in Kibana.

Here an example of my log file:

> 10.10.10.10 - - [17/Oct/2017:03:48:00 +0200] "GET /myurl/40 HTTP/1.1" 200 75793 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "0.277"

the last "0.277" were not in Kibana maybe because it's not following the standard format pattern.

According on what I red here an there the file format is defined in the following pipline:  
GET \_ingest/pipeline/filebeat-5.6.2-nginx-access-default

So I tried to update but didn't see any change:

```
"patterns": [
            """"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \[%{HTTPDATE:nginx.access.time}\] "%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} "%{DATA:nginx.access.referrer}" "%{DATA:nginx.access.agent}" "%{NUMBER:nginx.request_time}" """
          ]

```

I added the "%{NUMBER:nginx.request\_time}" at the end

According to the documentation about the pipeline:

> **[Create or update pipeline API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-pipeline-api.html)**

> ...pipeline changes take effect immediately.

So I expect to have nothing else to do??!!

Not related but when I edited the pipline to update it. Kibana return a syntax error, and I need to add a space before the last 3 """, if it can help somebody.

I also refresh the fields in Kibana but didn't see any new field.

Thanks for any help!

---

<div class="post-metadata">

**Author:** ![gabriel\_tessier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabriel_tessier/32/27911_2.png) [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)\
**Post date:** [October 18, 2017, 1:46am UTC](https://discuss.elastic.co/t/adding-more-fields-from-the-nginx-log-file/104231/2 "2017-10-18T01:46:38Z")

</div>

Got it!!

There's was a problem in the definition of the patterns from the nginx conf file:

```
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$request_time"';

```

The last $request\_time if you surround with " you need to put a space, that's why in Kibana it was complaining.

I solve it by removing the " around the $request\_time in the xginx conf file, but if you need to put " you need to put a space after.

All this can be tested with this command:

```
POST _ingest/pipeline/filebeat-5.6.2-nginx-access-default/_simulate
{
"docs":[
  {
    "_source": {
      "message": "10.10.10.10 - - [17/Oct/2017:03:48:00 +0200] \"GET /my_page/40 HTTP/1.1\" 200 75793 \"-\" \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\" 0.277"
    }
  }
  ]
}

```

But need to fix the patterns first by removing the " or adding space.

Thank you Elastic guys and girls to make so easy to use and good product!  
Sorry for the noise but hope that it can help somebody else.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2017, 1:46am UTC](https://discuss.elastic.co/t/adding-more-fields-from-the-nginx-log-file/104231/3 "2017-11-15T01:46:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.