# Logstash is Parsing Nginx log only half of it

**URL:** <https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733>\
**Category:** Logstash\
**Created:** [April 19, 2020, 1:52pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733 "2020-04-19T13:52:41Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 19, 2020, 1:52pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/1 "2020-04-19T13:52:42Z")

</div>

I have the following log comming from NGINX as reverse proxy set up.

Message:

> 192.168.1.24 - - [19/Apr/2020:15:39:03 +0200] "GET /website/static/src/scss/options/colors/website.assets\_wysiwyg/user\_theme\_color\_palette.scss.css HTTP/1.0" 304 0 "[https://maindomain.ch/impressum](https://maindomain.ch/impressum)" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 **(KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"**

The last part (bold) is not parsed into fields. There is one more problem. The referrer field ist including the " " so I have this "[https://maindomain.ch/impressum](https://maindomain.ch/impressum)" in the field. How can i fix that?

This is the logstash filter:

```
if [host][name] == "SVGXXX-XXXX-01.maindomain.ch" {
  if [event][module] == "nginx" {
    if [fileset][name] == "access" {
       mutate {
       add_tag => ["anginx", "Anginx"]
      }
      if "anginx" in [tags] {
      grok {
        match => **{ "message" => "%{COMBINEDAPACHELOG}+%{(?:"(?:%{URI:referrer}|-)"|%{QS:referrer})}+%{GREEDYDATA:extra_fields}" }**
       # remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
       }
      }
     }
    }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 3:24pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/2 "2020-04-19T15:24:19Z")

</div>

> [@hispeed](#):
>
> "%{COMBINEDAPACHELOG}+%{(?:"(?:%{URI:referrer}|-)"|%{QS:referrer})}+%{GREEDYDATA:extra\_fields}"

Why not just use HTTPD\_COMBINEDLOG, which includes quoted strings for agent and referrer? QS always includes the quotes in the match, so you would need to define your own patterns to avoid this.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 19, 2020, 3:51pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/3 "2020-04-19T15:51:39Z")

</div>

When I use HTTPD\_COMBINEDLOG it doesn't change anything. I tried different variants now.

Ok, can you help with creating my own patterns?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 4:47pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/4 "2020-04-19T16:47:05Z")

</div>

HTTPD\_COMBINEDLOG is

```
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}

```

Just remove the

```
+%{(?:"(?:%{URI:referrer}|-)"|%{QS:referrer})}+%{GREEDYDATA:extra_fields}

```

from your pattern. QS is rather complicated

```
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))

```

It handles both single and double quotes, there is a negative lookahead to avoid matching an opening double quote that is escaped, and there are several atomic matches that may or may not be performance optimizations. You could change it to avoid including the quotes in result (by moving the two outermost parentheses into each of the four alternated patterns) but it would be way easier to use

```
mutate { gsub => ["referrer", '^"', '', "referrer", '"$', ''] }
```

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 19, 2020, 5:00pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/5 "2020-04-19T17:00:10Z")

</div>

When I have it like this:

> if [event][module] == "nginx" {  
> if [fileset][name] == "access" {  
> mutate {  
> add\_tag =\> ["anginx", "Anginx"]  
> }  
> if "anginx" in [tags] {  
> grok {  
> match =\> { "message" =\> "%{HTTPD\_COMBINEDLOG}+%{HTTPD\_COMMONLOG}+%{QS:referrer}+%{QS:agent}" }  
> # remove\_field =\> "message"  
> }  
> mutate {  
> add\_field =\> { "read\_timestamp" =\> "%{@timestamp}" }  
> }

No field ar matched in Kibana. The mutate I would ad like this?

> if [event][module] == "nginx" {  
> if [fileset][name] == "access" {  
> mutate {  
> add\_tag =\> ["anginx", "Anginx"]  
> }  
> if "anginx" in [tags] {  
> grok {  
> match =\> { "message" =\> "%{HTTPD\_COMBINEDLOG}+%{HTTPD\_COMMONLOG}+%{QS:referrer}+%{QS:agent}" }  
> # remove\_field =\> "message"  
> }  
> mutate {  
> gsub =\> ["referrer", '^"', '', "referrer", '"$', '']  
> }  
> mutate {  
> add\_field =\> { "read\_timestamp" =\> "%{@timestamp}" }  
> }

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 5:11pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/6 "2020-04-19T17:11:27Z")

</div>

> [@hispeed](#):
>
> ```
> "%{HTTPD_COMBINEDLOG}+%{HTTPD_COMMONLOG}+%{QS:referrer}+%{QS:agent}"
> 
> ```

Your pattern requires that the message field contains one or more HTTPD\_COMBINEDLOG followed by one of more HTTPD\_COMMONLOG followed by one or more quoted strings followed by one or more quoted strings. I would not expect that to match the example message you quoted.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 19, 2020, 7:30pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/7 "2020-04-19T19:30:19Z")

</div>

I have now the following patterns. The referrer field is now without " " this works fine, thank you.  
But %{QS:referrer} %{QS:agent} doesn't work.

When I just use: HTTPD\_COMBINEDLOG %{HTTPD\_COMMONLOG} %{QS:referrer} %{QS:agent} nothing is parsed.

> if [event][module] == "nginx" {  
> if [fileset][name] == "access" {  
> mutate {  
> add\_tag =\> ["anginx", "Anginx"]  
> }  
> if "anginx" in [tags] {  
> grok {  
> match =\> { "message" =\> '%{HTTPD\_COMMONLOG} %{QS:referrer} %{QS:agent}' }  
> # remove\_field =\> "message"  
> }  
> mutate {  
> gsub =\> ["referrer", '^"', '', "referrer", '"$', '']  
> }  
> mutate {  
> add\_field =\> { "read\_timestamp" =\> "%{@timestamp}" }  
> }

i'm sorry I'm still a newbie concerning grok patterns. I have to see it how it works and then I study it and hope I understand it.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 7:58pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/8 "2020-04-19T19:58:29Z")

</div>

> [@hispeed](#):
>
> But %{QS:referrer} %{QS:agent} doesn't work.

What do you mean by that?

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 19, 2020, 8:02pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/9 "2020-04-19T20:02:43Z")

</div>

The message between the arrows is not splitted up into fields.

"referrer" is fine, sorry this was my mistake.

 ![not_parsed](https://us1.discourse-cdn.com/elastic/original/3X/9/a/9a0c54dbe72cb5ac7857ab2d693d91323c590ca0.jpeg)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 19, 2020, 11:49pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/10 "2020-04-19T23:49:20Z")

</div>

If you expect the agent field to be parsed you will need to add a [useragent](https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html) filter.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 20, 2020, 4:42am UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/11 "2020-04-20T04:42:02Z")

</div>

This is probably the solution, that I first need to install a useragent filter.

I don't have this right now. Which one do I need?

Is this version working in logstash 7.5.X and 7.6.X?

> **[logstash-plugins/logstash-filter-useragent](https://github.com/logstash-plugins/logstash-filter-useragent/tree/v3.2.4)**
>
> Contribute to logstash-plugins/logstash-filter-useragent development by creating an account on GitHub.

I have centos 8 installed, this shouldn't be a problem?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 20, 2020, 2:40pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/12 "2020-04-20T14:40:58Z")

</div>

> [@hispeed](#):
>
> I first need to install a useragent filter

It ships with the default logstash package, so you do not need to install it.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 20, 2020, 3:35pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/13 "2020-04-20T15:35:43Z")

</div>

Hi Badger ,

Ok I think I understand it maybe (50%). So yes I have already useragent in my patterns. Probably I just need to modify those lines?

I'm exactly at the same point: [Want a Logstash Filter to Parse the Agent Field in Apache Access Log Files](https://discuss.elastic.co/t/want-a-logstash-filter-to-parse-the-agent-field-in-apache-access-log-files/28387/5)

```
if [host][name] == "SVGXXX-XXXXX-XX.maindomain.ch" {
  if [event][module] == "nginx" {
    if [fileset][name] == "access" {
       mutate {
       add_tag => ["anginx", "Anginx"]
      }
      if "anginx" in [tags] {
      grok {
        match => { "message" => "%{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}" }
       # remove_field => "message"
      }
      mutate {
      gsub => ["referrer", '^"', '', "referrer", '"$', '']
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
       }
      }
     }
    }
    if [fileset][name] == "error" {
      grok {
        match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%
{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
        remove_field => "message"
      }
      mutate {
        rename => { "@timestamp" => "read_timestamp" }
      }
      date {
        match => ["[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
        remove_field => "[nginx][error][time]"
      }
    }
}

```

I recieve this:

`[2020-04-20T20:19:06,289][ERROR][logstash.filters.useragent][main] Uknown error while parsing user agent data {:exception=>#<TypeError: cannot convert instance of class org.jruby.RubyHash to class java.lang.String>, :field=>"[agent]", :event=>#<LogStash::Event:0x8da38e>}`

I also found this:

> **[Parse user agent strings into structured data based on BrowserScope data with...](https://cinhtau.net/2016/12/04/parse-user-agent-strings-into-structured-data-based-on-browserscope-data-with-logstash/)**
>
> The Apache HTTP Server logs user agent strings. The user agent string contains information like family, operating system, version, and device. Logstash offer...

At the moment, I didn't found any working solution.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 20, 2020, 6:41pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/14 "2020-04-20T18:41:01Z")

</div>

Now i'm a little bit more at the goal with this config:

```
  if [event][module] == "nginx" {
    if [fileset][name] == "access" {
       mutate {
       add_tag => ["anginx", "Anginx"]
      }
      if "anginx" in [tags] {
      grok {
        match => { "message" => "%{HTTPD_COMMONLOG} %{QS:referrer} %{QS:user_agent}" }
       # remove_field => "message"
      }
      mutate {
      gsub => ["referrer", '^"', '', "referrer", '"$', '']
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => ["[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][user_agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][user_agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
       }
      }
     }

```

I have now this in one field called: user agent:  
**"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"**

Now I want to split that up. Of course I will have different information inside and not always Windows NT 10.0 or Chrome.

---

<div class="post-metadata">

**Author:** ![sunnywilson09](https://avatars.discourse-cdn.com/v4/letter/s/7c8e57/32.png) [@sunnywilson09](https://discuss.elastic.co/u/sunnywilson09)\
**Post date:** [April 20, 2020, 7:02pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/15 "2020-04-20T19:02:09Z")

</div>

When I use HTTPD\_COMBINEDLOG it doesn't change anything. I tried different variants now.

---

<div class="post-metadata">

**Author:** ![hispeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hispeed/32/16232_2.png) [@hispeed](https://discuss.elastic.co/u/hispeed)\
**Post date:** [April 20, 2020, 7:06pm UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/16 "2020-04-20T19:06:51Z")

</div>

@sunnywilson09 what do you mean?  
This has nothing to do with my problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 19, 2020, 11:08am UTC](https://discuss.elastic.co/t/logstash-is-parsing-nginx-log-only-half-of-it/228733/18 "2020-05-19T11:08:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.