# How do I write parsers for multiple lines like this?

**URL:** <https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887>\
**Category:** Logstash\
**Created:** [July 27, 2018, 7:09am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887 "2018-07-27T07:09:25Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 27, 2018, 7:09am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/1 "2018-07-27T07:09:26Z")

</div>

Hi Team,

Can someone please help me here for creating parsers? Well my main confusion is and just need a hint how do I parse those multiple lines?

########################  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] connect  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3366] [192.168.44.1:60664] connect  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: GET /test/test.exe HTTP/1.1  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3367] [192.168.44.1:60665] connect  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: Host: 192.168.44.44  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: Connection: keep-alive  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: Upgrade-Insecure-Requests: 1  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,_/_;q=0.8  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: Accept-Encoding: gzip, deflate  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: Accept-Language: en-US,en;q=0.9,mr;q=0.8  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] info: Request URL: [http://192.168.44.44/test/test.exe](http://192.168.44.44/test/test.exe)  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] info: Sending fake file configured for extension 'exe'.  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] send: HTTP/1.1 200 OK  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] send: Content-Length: 24576  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] send: Content-Type: x-msdos-program  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] send: Date: Fri, 27 Jul 2018 07:22:33 GMT  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] send: Connection: Close  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] send: Server: Microsoft-IIS/6.0  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] info: Sending file: /var/lib/inetsim/http/fakefiles/sample\_gui.exe  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] stat: 1 method=GET url=http://192.168.44.44/test/test.exe sent=/var/lib/inetsim/http/fakefiles/sample\_gui.exe postdata=  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] disconnect

########################

How do I start parsing those logs then? Like Contain words recv, send, info?

TIA  
Blason R

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [July 27, 2018, 8:57am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/2 "2018-07-27T08:57:11Z")

</div>

Hi Blason,

Why don't you break this down? For example, you can write a grok similar to this:

```auto
\[%{DATA:datetime}\] \[%{DATA:id}\] \[%{DATA:policy}\] \[%{HOSTPORT:ipport}\] %{WORD:action}: %{GREEDYDATA:extra}

```

Then you can parse the value in the `extra` field using a combination of either conditionals or kv. Let me know if this works!

> PS: If the initial field format is fixed, using dissect would give you better performance. I have written it in Grok as i was not sure.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 27, 2018, 11:51am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/3 "2018-07-27T11:51:44Z")

</div>

hi @nerdsec thanks for the reply well I am only interested in certain lines and not all the lines, specifically

[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] recv: GET /test/test.exe HTTP/1.1  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] info: Request URL: [http://192.168.44.44/test/test.exe](http://192.168.44.44/test/test.exe)  
[2018-07-27 12:52:33] [3307] [http\_80\_tcp 3365] [192.168.44.1:60663] stat: 1 method=GET url=http://192.168.44.44/test/test.exe sent=/var/lib/inetsim/http/fakefiles/sample\_gui.exe postdata=

So in logstash how do I start the parsers so it will match lines containing recv and info? and then parameters of those? That is the confusing part for me.

Had it been a single line I would have written a single statment but since its different line not sure how do I start which has word recv and info.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [July 27, 2018, 12:32pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/4 "2018-07-27T12:32:38Z")

</div>

You can use a conditional to check if a value exists in the message field:

```auto
if [message] =~ /.*\s(recv:|info:|stat:)\s.*/ {
  parse the data
}
else
{
  drop {}
}

```

More details about this are here:  
[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html)

Also, you could actually use the parser i mentioned earlier and then apply a conditional. Not sure, which is more efficient. Maybe someone else can help for the efficiency.

```auto
if [action] in ["stat", "info", "recv"] {
  do stuff
}
else {
  drop {}
}
```

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [July 27, 2018, 12:37pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/5 "2018-07-27T12:37:02Z")

</div>

> [@Blason](#):
>
> Had it been a single line I would have written a single statment but since its different line not sure how do I start which has word recv and info.

I am sorry, i am not quite sure what you mean by this. Maybe an example would help.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 27, 2018, 12:54pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/6 "2018-07-27T12:54:20Z")

</div>

Hi,

above example where message contains is the correct one I believe and what I mean by single line is - if that was a single unform logs I could have parsed those but since all those were different lines was not sure how to pass those.

Let me try parsing the data that way and see.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [July 28, 2018, 5:19pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/7 "2018-07-28T17:19:13Z")

</div>

Sure! Let me know if you need any help. 🙂

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [July 31, 2018, 5:59am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/8 "2018-07-31T05:59:37Z")

</div>

Well I did configure the parsers for the same and here are those however after that I am only interested in certain lines and not all but dont have a clue how to proceed?

filter {

```
    grok {
            match => { "message" => "(\[%{TIMESTAMP_ISO8601:logdate}\] \[%{DATA:pid}\] \[%{DATA:port} %{DATA:data}\] \[%{IPV4:clientipaddr}\:%{WORD:sport}\] %{WORD:action}\: %{GREEDYDATA:rest_message})"
               }
        }

```

}

output {

```
    elasticsearch {
            index => "logstash-deception-%{+YYYY.MM.dd}"
            hosts => ["http://localhost:9200"]
            }
    }

```

So these are matching the logs till and after that its just a GREEDYDATA; i wanted to fitler certain lines from those and parse those.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [July 31, 2018, 6:15pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/9 "2018-07-31T18:15:04Z")

</div>

Considering the grok expression you have written, a basic data structure would be as follows:

```auto
{
  "data": "3365",
  "port": "http_80_tcp",
  "clientipaddr": "192.168.44.1",
  "logdate": "2018-07-27 12:52:33",
  "action": "recv",
  "pid": "3307",
  "rest_message": "GET /test/test.exe HTTP/1.1",
  "sport": "60663"
}

```

Considering the sample logs that you have provided, the information in the `rest_message` would not be consistent across multiple lines as the string has different values based on the context. But looking closely, there seems to be key-value pairing between the values (based on ":"). Would using a `kv` filter solve the issue? Here is an example:

```auto
filter {
  kv {
    value_split => ":"
  }
}

```

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html)

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [July 31, 2018, 6:17pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/10 "2018-07-31T18:17:08Z")

</div>

> [@NerdSec](#):
>
> GET /test/test.exe HTTP/1.1

The above mentioned logic would fail only for string with the response codes and URI requested messages. Are there any other such scenarios in your logs? You could use conditionals to find out such scenarios and take appropriate actions.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 1, 2018, 2:00pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/11 "2018-08-01T14:00:14Z")

</div>

Hi there,

Thanks for the pointer and here are the complete logs from connect TO disconnect. I am only interested in certain lines may be the logs which contains Info: and then parse those.  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] connect  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: GET /test1/test2/test3.pdf HTTP/1.1  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: Host: 000cas.info  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: Connection: keep-alive  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: Upgrade-Insecure-Requests: 1  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko  
) Chrome/67.0.3396.99 Safari/537.36  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,_/_;q=0  
.8  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: Accept-Encoding: gzip, deflate, br  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] recv: Accept-Language: en-US,en;q=0.9  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] info: Request URL: [https://000cas.info/test1/test2/test3.pdf](https://000cas.info/test1/test2/test3.pdf)  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] info: Sending fake file configured for extension 'pdf'.  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] send: HTTP/1.1 200 OK  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] send: Content-Type: application/pdf  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] send: Server: Microsoft-IIS/8.0  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] send: Date: Tue, 31 Jul 2018 13:07:35 GMT  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] send: Content-Length: 37053  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] send: Connection: Close  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] info: Sending file: /var/lib/inetsim/http/fakefiles/sample.pdf  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] stat: 1 method=GET url=https://000cas.info/test1/test2/test3.pdf sent=/var/lib/inetsim/http/fakef  
iles/sample.pdf postdata=  
[2018-07-31 18:37:35] [31771] [https\_443\_tcp 704] [192.168.1.33:27025] disconnect

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [August 1, 2018, 5:29pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/12 "2018-08-01T17:29:33Z")

</div>

Well in that case, I would say you have already parsed your logs, right?

The data after info field is quite generic and can be kept as it is.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 1, 2018, 5:37pm UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/13 "2018-08-01T17:37:27Z")

</div>

Yep right!! But keen to know I need logs only from Info: field rest everything should be dropped.

---

<div class="post-metadata">

**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\
**Post date:** [August 2, 2018, 4:10am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/14 "2018-08-02T04:10:22Z")

</div>

> [@NerdSec](#):
>
> Also, you could actually use the parser i mentioned earlier and then apply a conditional. Not sure, which is more efficient. Maybe someone else can help for the efficiency.
> 
> ```auto
> if [action] in ["stat", "info", "recv"] {
> do stuff
> }
> else {
> drop {}
> }
> 
> ```

Refer this post. Remove the stat and recv part, and keep only info.

The drop filter will remove all logs that you do not need.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [August 5, 2018, 7:28am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/15 "2018-08-05T07:28:24Z")

</div>

great thnks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 2, 2018, 7:28am UTC](https://discuss.elastic.co/t/how-do-i-write-parsers-for-multiple-lines-like-this/141887/16 "2018-09-02T07:28:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.