Parsing a lot of different syslogs using logstash

Elastic Stack Logstash
1

Hi guys!

I have to parse a lot of different syslogs. Here are a few examples:

07/31/2018:08:45:02 GMT res-nsvpx-pri 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 15132959 0 : SPCBId 483501867 - ClientIP 40.115.27.193 - ClientPort 53758 - VserverServiceIP 10.3.49.21 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - Session New

07/31/2018:08:42:48 GMT xer-p-nsvpx01 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_FAILURE 18542698 0 : SPCBId 881186233 - ClientIP 40.115.125.114 - ClientPort 52241 - VserverServiceIP 10.3.40.135 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - Reason "No matching certificate found"

07/31/2018:07:05:31 GMT wgkp-nsvpx02 0-PPE-0 : default AAATM LOGIN 4003109 0 : Context lcasteleyn@213.118.59.101 - SessionId: 19354- User lcasteleyn - Client_ip 213.118.59.101 - Nat_ip "Mapped Ip" - Vserver 10.3.172.22:443 - Browser_type "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" - Group(s) "N/A"

The first part (date, time, timezone, hostname and packet engine) are always the same in each syslog. I have tried to parse this part and then start to parse second part that is not always the same, but it didn't work with my code:

else if "nsvpx" in [tags]{
   grok{
       match => {"message" => "%{DATE:date}:%{HAPROXYTIME:time} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{GREEDYDATA:message}"}
       mutate {remove_field => ["date", "time", "timezone", "haproxy_hour",  "haproxy_minute", "haproxy_second"]}
       overwrite => ["message"]
       tag_on_failure => ["_grokparsefailure_nsvpx","_grokparsefailure"]
    }
}

I got this back with 1 extra field "hostname:15132959" :

SPCBId 483501867 - ClientIP 40.115.27.193 - ClientPort 53758 - VserverServiceIP 10.3.49.21 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "AES-256-CBC-SHA TLSv1.2 Non-Export 256-bit" - Session New

I've tried to parse whole message each time, but it didn't work. Here is my code for 3 examples above:

 else if "nsvpx" in [tags]{
     grok{
 		break_on_match => true 
		match => [
			"message", "%{DATE:datum}:%{HAPROXYTIME:tijd} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : (?<event> default SSLLOG SSL_HANDSHAKE_SUCCESS) %{GREEDYDATA} - ClientIP %{IP:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IP:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Session %{WORD:Session}",
			"message", "%{DATE:datum}:%{HAPROXYTIME:tijd} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : (?<event> default SSLLOG SSL_HANDSHAKE_FAILURE) %{GREEDYDATA} - ClientIP %{IP:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IP:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}",
			"message", "%{DATE:datum}:%{HAPROXYTIME:tijd} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{WORD:event1} %{WORD:event2} %{WORD:event3} %{GREEDYDATA} : Context %{GREEDYDATA} User %{WORD:user} - Client_ip %{IP:ClientIP} - Nat_ip %{QS:natIP} - Vserver %{IP:VserverIP}:%{NUMBER:VserverPort} - Browser_type %{QS:browser_type} - Group\(s\) %{QS:groups}"
		]
    }
}

Now I'm not using the overwrite command, I know that it was wrong.

Is there a way to escape double quote and slash and let them be a part of the field like in this examples:

Command "login rancid "password"" (instead of word password it has to be all stars like below)
it has to return : login rancid "********" and Command %{QS:command} doesn't work

User wvanhoucke : Group(s) N/A : Vserver 10.3.172.22:443
it has to return: Group(s) : N/A and I have no idea how to do that.

Is there a way to parse al those messages correctly?

NOTE: everything below is not a problem anymore

(I have there about 40 different syslog messages and each message can also contain several differences)

For example I have these 2 messages:
31/07/2018:11:04:53 GMT WIV-P-NSVPX01 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_FAILURE 31237256 0 : SPCBId 28317873 - ClientIP 35.202.99.105 - ClientPort 57918 - VserverServiceIP 10.72.128.37 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "NA" - Reason "No shared cipher"

07/31/2018:10:06:32 GMT SVHG-NSVPX01 0-PPE-0 : SSLLOG SSL_HANDSHAKE_FAILURE 906645 0 : SPCBId 10842232 - ClientIP 10.2.0.119 - ClientPort 5326 - VserverServiceIP 10.130.1.9 - VserverServicePort 443 - ClientVersion TLSv1.0 - CipherSuite "DES-CBC3-SHA TLSv1 Non-Export 168-bit" - Reason "Fatal alert received from the client"

Single difference is the word "default" in the message. I've used this code, but it didn't work:

else if "nsvpx" in [tags]{
if "default" in [message]{
  grok{
      break_on_match => true 
  	match => [
    	"message", "%{DATE:date}:%{HAPROXYTIME:time} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{WORD:event1} %{WORD:event2} %{WORD:event3} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} :  SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"
      ]
        tag_on_failure => ["_grokparsefailure_nsvpx_parsing"]
  }
}
else{
  grok{
      break_on_match => true 
  	match => [
    	"message", "%{DATE:date}:%{HAPROXYTIME:time} %{WORD:timezone} %{USERNAME:hostname} %{USERNAME:packet_engine} : %{WORD:event1} %{WORD:event2} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} :  SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"
      ]
        tag_on_failure => ["_grokparsefailure_nsvpx_parsing"]
  }
}

}

Thanks a lot!

2

That is a lot of different questions in one post!

I would start by parsing off the common fields using a dissect filter.

dissect { mapping => { "message" => "%{ts} %{+ts} %{hostname} %{packet-engine} : %{restOfLine}" } }

You appear to be using mutate to discard the timestamp, if you really do not want those fields then you can use

 dissect { mapping => { "message" => "%{} %{} %{hostname} %{packet-engine} : %{restOfLine}" } }
3

Thank you for your reply!

Did I understand you right, @Badger ? (because it doesn't work :slightly_frowning_face:)

else if "nsvpx" in [tags]{
    dissect{
		break_on_match => true 
	   	mapping => [
		  "message", "%{}:%{} %{} %{hostname} %{packet_engine} : %{event1} %{event2} %{event3} %{} %{} :  SPCBId %{} - ClientIP %{ClientIP} - ClientPort %{ClientPort} - VserverServiceIP %{VserverIP} - VserverServicePort %{VserverPort} - ClientVersion %{ClientVersion} - CipherSuite %{CipherSuite} - Session %{Session}",
		  "message", "%{}:%{} %{} %{hostname} %{packet_engine} : %{event1} %{event2} %{event3} %{} %{} :  Backend SPCBId %{} - ServerIP %{ServerIP} - ServerPort %{ServerPort} - ProtocolVersion %{ProtocolVersion} - CipherSuite %{CipherSuite} - Session %{Session} - SERVER_AUTHENTICATED -SerialNumber %{serial_number} - SignatureAlgorithm %{signature_algorithm} - ValidFrom %{valid_from} - ValidTo %{valid_to} - HandshakeTime %{handshaketime_ms} %{}",
		  "message", "%{}:%{} %{} %{hostname} %{packet_engine} : %{event1} %{event2} %{event3} %{} %{} :  SPCBId %{} - ClientIP %{ClientIP} - ClientPort %{ClientPort} - VserverServiceIP %{VserverIP} - VserverServicePort %{VserverPort} - ClientVersion %{ClientVersion} - CipherSuite %{CipherSuite} - Reason %{reason}"
		  ]
        tag_on_failure => ["_grokparsefailure_nsvpx_parsing"]
    }
 }
4

No, use the filter just the way I wrote it, then use grok to parse out all the different values for the restOfLine field.

Note that your patterns are not anchored (they do not start with ^ to anchor them to the start of the field or $ to anchor them to the end). That means that

"%{WORD:event2} %{WORD:event3} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} : SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}"

matches both of these lines

default SSLLOG SSL_HANDSHAKE_FAILURE 31237256 0 : SPCBId 28317873 - ClientIP 35.202.99.105 - ClientPort 57918 - VserverServiceIP 10.72.128.37 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "NA" - Reason "No shared cipher"
SSLLOG SSL_HANDSHAKE_FAILURE 906645 0 : SPCBId 10842232 - ClientIP 10.2.0.119 - ClientPort 5326 - VserverServiceIP 10.130.1.9 - VserverServicePort 443 - ClientVersion TLSv1.0 - CipherSuite "DES-CBC3-SHA TLSv1 Non-Export 168-bit" - Reason "Fatal alert received from the client"

So those log messages would be parsed by

    dissect { mapping => { "message" => "%{ts} %{+ts} %{hostname} %{packet-engine} : %{restOfLine}" } }
    grok { match => { "restOfLine" => [ "%{WORD:event2} %{WORD:event3} %{BASE16NUM:randomnr} %{NUMBER:randomnr2} : SPCBId %{BASE16NUM:SPCBId} - ClientIP %{IPV4:ClientIP} - ClientPort %{NUMBER:ClientPort} - VserverServiceIP %{IPV4:VserverIP} - VserverServicePort %{NUMBER:VserverPort} - ClientVersion %{URIHOST:ClientVersion} - CipherSuite %{QS:CipherSuite} - Reason %{QS:reason}" ] } }
5

Without If else condition it's impossible to distinguish the difference between messages with/without "default" word.
I don't why, but my code in the 1st post works now correctly, but thank you very much for your help, I've learned some new things!

with%20default
with default.PNG1506×252 22.9 KB

6

Yeah, but you can use something like

if [restOfLine] =~ /^default/ {
    mutate { add_tag => [ "default" ] }
}

to distinguish those two.

1 Like
7

thank you! I will try it!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.