Lag in logs

[kriti_dabas]

filter {
  json {
    source => "message"
  }
  
  grok {
    match => {
      "message" => [
         "%{DATA:description} default %{DATA:connection_details} :  SPCBId %{DATA:spcbId} - ClientIP %{DATA:clientIP} - ClientPort %{DATA:clientPort} - VserverServiceIP %{DATA:vserverServiceIP} - VserverServicePort %{DATA:vserverServicePort} - ClientVersion %{DATA:clientVersion} - CipherSuite %{DATA:cipherSuite} - Session %{DATA:SessionStatus} - HandshakeTime %{DATA:handshakeTime} ms",
         "%{DATA:description} default %{DATA:connection_details} :  SPCBId %{DATA:spcbId} - ClientIP %{DATA:clientIP} - ClientPort %{DATA:clientPort} - VserverServiceIP %{DATA:vserverServiceIP} - VserverServicePort %{DATA:vserverServicePort} - ClientVersion %{DATA:clientVersion} - CipherSuite \"%{DATA:cipherSuite}\"Session %{DATA:SessionStatus} - Reason \"%{DATA:reason}\"",
         "%{DATA:description} default %{DATA:connection_details} Source %{IP:SourceIP}:%{NUMBER:SourcePort} - Destination %{IP:DestinationIP}:%{NUMBER:DestinationPort} - Start Time %{DATA:Starttime} - End Time %{DATA:Endtime} %{GREEDYDATA:Total_bytes_recv_snd}",
         "%{DATA:description} default %{DATA:connection_details} Source %{IP:SourceIP}:%{NUMBER:SourcePort} - Vserver %{IP:VserverIP}:%{NUMBER:VserverPort} - NatIP %{DATA:NatIP}:%{NUMBER:NatIPPort} - Destination %{IP:DestinationIP}:%{NUMBER:DestinationPort} - Delink Time %{DATA:DelinkTime} %{DATA:Total_bytes_recv_snd}",
         "%{DATA:description} default %{DATA:connection_details}  Source %{IP:SourceIP}:%{NUMBER:SourcePort} - Destination %{IP:Destination1IP}:%{NUMBER:Destination1Port} - NatIP %{DATA:NatIP}:%{NUMBER:NatIPPort} - Destination %{IP:Destination2IP}:%{NUMBER:Destination2Port} - Start Time %{DATA:Starttime} - Delink Time %{DATA:DelinkTime} %{GREEDYDATA:Total_bytes_recv_snd}(?= - Closure Reason) - %{GREEDYDATA:Reason}",
         "%{DATA:description} : default %{GREEDYDATA:connection_details} - %{DATA:Profile} %{DATA:Action} %{URI:url} %{GREEDYDATA:Status}"
    ]
    }
  }
}

because of the use of this filter my cpu usage on logstash is increasing due to which there is a lag in my logs so how to simplify my filter ?

[leandrojmp (Leandro Pereira)]

Grok can be CPU intensive, and your grok filter is not optimized and probably can be improved.

I recommend reading this blog post about grok.

Also, can you share sample messages of each one of your grok patterns so people can try to replicate your filter and see how to optimize it?

To have some suggestions on how to optimize it you would need to share some sample messages.

[miltonhultgren]

Before you try to optimize your Grok patterns, look if you can get by with other cheaper processor like dissect.