# Parsing a lot of different syslogs using logstash

**URL:** <https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324>\
**Category:** Logstash\
**Created:** [July 31, 2018, 9:33am UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324 "2018-07-31T09:33:04Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kateryna](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@Kateryna](https://discuss.elastic.co/u/Kateryna)\
**Post date:** [July 31, 2018, 9:33am UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/1 "2018-07-31T09:33:05Z")

</div>

Hi guys!

I have to parse a lot of different syslogs. Here are a few examples:

> 07/31/2018:08:45:02 GMT res-nsvpx-pri 0-PPE-0 : default SSLLOG SSL\_HANDSHAKE\_SUCCESS 15132959 0 : SPCBId 483501867 - ClientIP 40.115.27.193 - ClientPort 53758 - VserverServiceIP 10.3.49.21 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - Session New

> 07/31/2018:08:42:48 GMT xer-p-nsvpx01 0-PPE-0 : default SSLLOG SSL\_HANDSHAKE\_FAILURE 18542698 0 : SPCBId 881186233 - ClientIP 40.115.125.114 - ClientPort 52241 - VserverServiceIP 10.3.40.135 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - Reason "No matching certificate found"

> 07/31/2018:07:05:31 GMT wgkp-nsvpx02 0-PPE-0 : default AAATM LOGIN 4003109 0 : Context [lcasteleyn@213.118.59.101](mailto:lcasteleyn@213.118.59.101) - SessionId: 19354- User lcasteleyn - Client\_ip 213.118.59.101 - Nat\_ip "Mapped Ip" - Vserver 10.3.172.22:443 - Browser\_type "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" - Group(s) "N/A"

The first part (date, time, timezone, hostname and packet engine) are always the same in each syslog. I have tried to parse this part and then start to parse second part that is not always the same, but it didn't work with my code:

> ```
> else if "nsvpx" in [tags]{
> grok{
> match => {"message" => "%{DATE:date}:%{HAPROXYTIME:time} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{GREEDYDATA:message}"}
> mutate {remove_field => ["date", "time", "timezone", "haproxy_hour", "haproxy_minute", "haproxy_second"]}
> overwrite => ["message"]
> tag_on_failure => ["_grokparsefailure_nsvpx","_grokparsefailure"]
> }
> }
> 
> ```

I got this back with 1 extra field "hostname:15132959" :

> SPCBId 483501867 - ClientIP 40.115.27.193 - ClientPort 53758 - VserverServiceIP 10.3.49.21 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - Session New

I've tried to parse whole message each time, but it didn't work. Here is my code for 3 examples above:

> ```
> else if "nsvpx" in [tags]{
> grok{
> break_on_match => true 
> match => [
> "message", "%{DATE:datum}:%{HAPROXYTIME:tijd} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : (?<event> default SSLLOG SSL_HANDSHAKE_SUCCESS) %{GREEDYDATA} - ClientIP %{IP:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IP:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Session %{WORD:Session}",
> "message", "%{DATE:datum}:%{HAPROXYTIME:tijd} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : (?<event> default SSLLOG SSL_HANDSHAKE_FAILURE) %{GREEDYDATA} - ClientIP %{IP:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IP:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}",
> "message", "%{DATE:datum}:%{HAPROXYTIME:tijd} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{WORD:event1} %{WORD:event2} %{WORD:event3} %{GREEDYDATA} : Context %{GREEDYDATA} User %{WORD:user} - Client_ip %{IP:ClientIP} - Nat_ip %{QS:natIP} - Vserver %{IP:VserverIP}:%{NUMBER:VserverPort} - Browser_type %{QS:browser_type} - Group\(s\) %{QS:groups}"
> ]
> }
> }
> 
> ```

Now I'm not using the overwrite command, I know that it was wrong.

Is there a way to escape double quote and slash and let them be a part of the field like in this examples:

> Command "login rancid "password"" (instead of word password it has to be all stars like below)  
> it has to return : login rancid "\*\*\*\*\*\*\*\*" and Command %{QS:command} doesn't work

> User wvanhoucke : Group(s) N/A : Vserver 10.3.172.22:443  
> it has to return: Group(s) : N/A and I have no idea how to do that.

Is there a way to parse al those messages correctly?

NOTE: everything below is not a problem anymore

> (I have there about 40 different syslog messages and each message can also contain several differences)
> 
> For example I have these 2 messages:  
> 31/07/2018:11:04:53 GMT WIV-P-NSVPX01 0-PPE-0 : default SSLLOG SSL\_HANDSHAKE\_FAILURE 31237256 0 : SPCBId 28317873 - ClientIP 35.202.99.105 - ClientPort 57918 - VserverServiceIP 10.72.128.37 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "NA" - Reason "No shared cipher"
> 
> 07/31/2018:10:06:32 GMT SVHG-NSVPX01 0-PPE-0 : SSLLOG SSL\_HANDSHAKE\_FAILURE 906645 0 : SPCBId 10842232 - ClientIP 10.2.0.119 - ClientPort 5326 - VserverServiceIP 10.130.1.9 - VserverServicePort 443 - ClientVersion TLSv1.0 - CipherSuite "DES-CBC3-SHA TLSv1 Non-Export 168-bit" - Reason "Fatal alert received from the client"
> 
> Single difference is the word "default" in the message. I've used this code, but it didn't work:
> 
> ```
> else if "nsvpx" in [tags]{
> if "default" in [message]{
> grok{
> break_on_match => true 
> match => [
> "message", "%{DATE:date}:%{HAPROXYTIME:time} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{WORD:event1} %{WORD:event2} %{WORD:event3} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} : SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"
> ]
> tag_on_failure => ["_grokparsefailure_nsvpx_parsing"]
> }
> }
> else{
> grok{
> break_on_match => true 
> match => [
> "message", "%{DATE:date}:%{HAPROXYTIME:time} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{WORD:event1} %{WORD:event2} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} : SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"
> ]
> tag_on_failure => ["_grokparsefailure_nsvpx_parsing"]
> }
> }
> 
> ```
> 
> }

Thanks a lot!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 31, 2018, 12:09pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/2 "2018-07-31T12:09:13Z")

</div>

That is a lot of different questions in one post!

I would start by parsing off the common fields using a dissect filter.

```
dissect { mapping => { "message" => "%{ts} %{+ts} %{hostname} %{packet-engine} : %{restOfLine}" } }

```

You appear to be using mutate to discard the timestamp, if you really do not want those fields then you can use

```
 dissect { mapping => { "message" => "%{} %{} %{hostname} %{packet-engine} : %{restOfLine}" } }
```

---

<div class="post-metadata">

**Author:** ![Kateryna](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@Kateryna](https://discuss.elastic.co/u/Kateryna)\
**Post date:** [July 31, 2018, 1:30pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/3 "2018-07-31T13:30:18Z")

</div>

Thank you for your reply!

Did I understand you right, @Badger ? (because it doesn't work 🙁)

> ```
> else if "nsvpx" in [tags]{
> dissect{
> break_on_match => true 
> mapping => [
> "message", "%{}:%{} %{} %{hostname} %{packet_engine} : %{event1} %{event2} %{event3} %{} %{} : SPCBId %{} - ClientIP %{ClientIP} - ClientPort %{ClientPort} - VserverServiceIP %{VserverIP} - VserverServicePort %{VserverPort} - ClientVersion %{ClientVersion} - CipherSuite %{CipherSuite} - Session %{Session}",
> "message", "%{}:%{} %{} %{hostname} %{packet_engine} : %{event1} %{event2} %{event3} %{} %{} : Backend SPCBId %{} - ServerIP %{ServerIP} - ServerPort %{ServerPort} - ProtocolVersion %{ProtocolVersion} - CipherSuite %{CipherSuite} - Session %{Session} - SERVER_AUTHENTICATED -SerialNumber %{serial_number} - SignatureAlgorithm %{signature_algorithm} - ValidFrom %{valid_from} - ValidTo %{valid_to} - HandshakeTime %{handshaketime_ms} %{}",
> "message", "%{}:%{} %{} %{hostname} %{packet_engine} : %{event1} %{event2} %{event3} %{} %{} : SPCBId %{} - ClientIP %{ClientIP} - ClientPort %{ClientPort} - VserverServiceIP %{VserverIP} - VserverServicePort %{VserverPort} - ClientVersion %{ClientVersion} - CipherSuite %{CipherSuite} - Reason %{reason}"
> ]
> tag_on_failure => ["_grokparsefailure_nsvpx_parsing"]
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 31, 2018, 1:48pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/4 "2018-07-31T13:48:29Z")

</div>

No, use the filter just the way I wrote it, then use grok to parse out all the different values for the restOfLine field.

Note that your patterns are not anchored (they do not start with ^ to anchor them to the start of the field or $ to anchor them to the end). That means that

```
"%{WORD:event2} %{WORD:event3} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} : SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"

```

matches both of these lines

```
default SSLLOG SSL_HANDSHAKE_FAILURE 31237256 0 : SPCBId 28317873 - ClientIP 35.202.99.105 - ClientPort 57918 - VserverServiceIP 10.72.128.37 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "NA" - Reason "No shared cipher"
SSLLOG SSL_HANDSHAKE_FAILURE 906645 0 : SPCBId 10842232 - ClientIP 10.2.0.119 - ClientPort 5326 - VserverServiceIP 10.130.1.9 - VserverServicePort 443 - ClientVersion TLSv1.0 - CipherSuite "DES-CBC3-SHA TLSv1 Non-Export 168-bit" - Reason "Fatal alert received from the client"

```

So those log messages would be parsed by

```
    dissect { mapping => { "message" => "%{ts} %{+ts} %{hostname} %{packet-engine} : %{restOfLine}" } }
    grok { match => { "restOfLine" => ["%{WORD:event2} %{WORD:event3} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} : SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"] } }
```

---

<div class="post-metadata">

**Author:** ![Kateryna](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@Kateryna](https://discuss.elastic.co/u/Kateryna)\
**Post date:** [July 31, 2018, 2:11pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/5 "2018-07-31T14:11:12Z")

</div>

Without If else condition it's impossible to distinguish the difference between messages with/without "default" word.  
I don't why, but my code in the 1st post works now correctly, but thank you very much for your help, I've learned some new things!

 ![with%20default](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b5c840bd8041ef6ccd4692aa49341f74540d0ca6.PNG)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 31, 2018, 2:21pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/6 "2018-07-31T14:21:43Z")

</div>

Yeah, but you can use something like

```
if [restOfLine] =~ /^default/ {
    mutate { add_tag => ["default"] }
}

```

to distinguish those two.

---

<div class="post-metadata">

**Author:** ![Kateryna](https://avatars.discourse-cdn.com/v4/letter/k/b5a626/32.png) [@Kateryna](https://discuss.elastic.co/u/Kateryna)\
**Post date:** [July 31, 2018, 2:24pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/7 "2018-07-31T14:24:50Z")

</div>

thank you! I will try it!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 28, 2018, 2:24pm UTC](https://discuss.elastic.co/t/parsing-a-lot-of-different-syslogs-using-logstash/142324/8 "2018-08-28T14:24:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.