# Grok different types of log in one file

**URL:** <https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015>\
**Category:** Logstash\
**Created:** [May 31, 2018, 10:05am UTC](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015 "2018-05-31T10:05:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kimfut](https://avatars.discourse-cdn.com/v4/letter/k/b9e5f3/32.png) [@kimfut](https://discuss.elastic.co/u/kimfut)\
**Post date:** [May 31, 2018, 10:05am UTC](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015/1 "2018-05-31T10:05:19Z")

</div>

hello every one , i have a file which contain 3 types of log :

1 )

> X.X.X.X - - [04/May/2018:16:24:21 +0200] "GET /WebServiceRequestProxyCenter/api/request-url?country=fr&url=https%3A%2F%[2Fwww.google.fr](http://2Fwww.google.fr)%2Fsearch%3Fq%3Disolation%2520thermique%26start%3D70%26ie%3DUTF-8%26oe%3DUTF8%26hl%3Dfr&service=new-rescan-rank%2C+siniat%2C+https%3A%2F%[2Fwww.siniat.fr](http://2Fwww.siniat.fr)%2C+isolation+thermique HTTP/1.1" 200 92866

1. 

> X.X.X.X - - [04/May/2018:15:57:26 +0200] "GET /WebServiceRequestProxyCenter/api/request-url?service=semantic\_analyze\_missing%2C+demo14%2C+friteuse+electrique+pas+cher&url=https%3A%2F%[2Fwww.google.fr](http://2Fwww.google.fr)%2Fsearch%3Fq%3Dsite%253Awww.friteuses.info%2Bfriteuse%2Belectrique%2Bpas%2Bcher%26ie%3DUTF-8%26oe%3DUTF8%26hl%3Dfr%26start%3D0%26num%3D10&country=fr HTTP/1.1" 200 104563

1. 

> X.X.X.X - - [04/May/2018:16:53:53 +0200] "GET /WebServiceRequestProxyCenter/monitor HTTP/1.1" 200 604

(X.X.X.X = it's an ip address )

for this time , i wrote a grok :

> filter {
> 
> ```
> if [clientip] == "X.X.X.X" {
> grok {
> 
> match => { "message" => ['%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request}\?%{DATA:request2}\&%{DATA:request3}HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})'] } } 
> 
> ```
> 
> }  
> if [clientip] == "X.X.X.X" {
> 
> grok {  
> match =\> { "message" =\> ['%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request}HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})']  
> }  
> }  
> }
> 
> }  
> grok {
> 
> match =\> { "message" =\> ['%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "%{WORD:verb} %{DATA:request}?%{DATA:request2}&%{DATA:request3}&%{DATA:request4}HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})']  
> }
> 
> }

but this grok it's something manual and the result it's not uniformed , for example :

result for log type 2)

 ![a](https://us1.discourse-cdn.com/elastic/original/3X/b/d/bd944d6a13834e229868ff136f4d705f89361fc2.png)

here we can see that country it's allowed for variable "request4"  
result for log type 1)

 ![b](https://us1.discourse-cdn.com/elastic/original/3X/7/9/794334e1e4ebe5b9d75c243135353f33e017fd94.png)

we can see that country it's allowed variable "request2"

so the question is : i want to know if there is an automatic method to extract and split from all my logs to get :  
request1 = /WebServiceRequestProxyCenter/api/request-url  
request2 = service  
request3 = url  
request4 = country  
?

---

<div class="post-metadata">

**Author:** ![kimfut](https://avatars.discourse-cdn.com/v4/letter/k/b9e5f3/32.png) [@kimfut](https://discuss.elastic.co/u/kimfut)\
**Post date:** [May 31, 2018, 3:18pm UTC](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015/2 "2018-05-31T15:18:15Z")

</div>

any help please ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 31, 2018, 5:30pm UTC](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015/4 "2018-05-31T17:30:06Z")

</div>

For something highly structured like a web server logs I would use dissect rather than grok. Personally I would probably mutate+split the query string to get an array, but...

```auto
    dissect { mapping => { "message" => '%{ip} %{ident} %{auth} [%{ts}] "%{method} %{uriAndQs} %{protocol}" %{status} %{bytes}' } }
    if [uriAndQs] !~ /\?/ {
        mutate { rename => { "uriAndQs" => "uri" } }
    } else {
        dissect { mapping => { "uriAndQs" => "%{uri}?{%{qs}" } }
        grok {
            match => { "qs" => [
                    "^(?<r1>[^&]+)&(?<r2>[^&]+)&(?<r3>[^&]+)$",
                    "^(?<r1>[^&]+)&(?<r2>[^&]+)$",
                    "^(?<r1>[^&]+)$"
                ]
            }
        }
    }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2018, 5:30pm UTC](https://discuss.elastic.co/t/grok-different-types-of-log-in-one-file/134015/5 "2018-06-28T17:30:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.