Parsing different lines in same log

Elastic Stack Logstash
1

Hi, I've been trying to parse a log for a couple of days now using Grok. I made it work and then, all of a sudden, it stopped working and I don't know why.

I've been using the grok debugger but I haven't been able to parse the log correctly.

This is the log I want to parse...

0|parity   | 2019-02-20 10:31:56 UTC Imported #1400403 0xb0cd…dae6 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:00 UTC Imported #1400404 0xdb1e…54ca (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:01 UTC    4/25 peers      4 MiB chain  180 MiB db  0 bytes queue   11 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
0|parity   | 2019-02-20 10:32:04 UTC Imported #1400405 0x38cd…4cfc (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:08 UTC Imported #1400406 0x11c2…67ab (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:12 UTC Imported #1400407 0xba99…10f2 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:16 UTC Imported #1400408 0x13b3…9bec (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:20 UTC Imported #1400409 0xa213…2978 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:24 UTC Imported #1400410 0xbf03…59d2 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:28 UTC Imported #1400411 0x155d…6da4 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:32 UTC Imported #1400412 0x537f…93b8 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:36 UTC Imported #1400413 0x3ea7…b66c (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:36 UTC    4/25 peers      4 MiB chain  180 MiB db  0 bytes queue   11 KiB sync  RPC:  0 conn,    0 req/s,    0 µs
0|parity   | 2019-02-20 10:32:40 UTC Imported #1400414 0x4b35…1507 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)
0|parity   | 2019-02-20 10:32:44 UTC Imported #1400415 0x06fb…d587 (0 txs, 0.00 Mgas, 0 ms, 0.57 KiB)

As you can see, there's two types of lines.
On the first ones, I want to extract the block number (after #) and the transactions (before txs).
On the second type of line, I want to extract the connected peers (the part before the /25 peers).

So far, my logstash configuration is like this:

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    # block message
    match => { "message" => "#%{NUMBER:block}.*%{NUMBER:txs} txs" }
    add_tag => ["blockMessage"]
  }

  grok {
    # peer message
    match => { "message" => ".* %{NUMBER:connectedPeers}/%{NUMBER:maxPeers} peers" }
    add_tag => ["peersMessage"]
  }

  mutate {
    convert => {
      "block" => "integer"
      "txs" => "integer"
      "connectedPeers" => "integer"
      "maxPeers" => "integer"
    }
  }
}

output {
   stdout { codec => rubydebug }
}

I've commented the mutate part, thinking that maybe Logstash was failing trying to convert, for example, txs on a connected peers line, but it still doesn't work.

¿Should I add some kind of conditional code related to each tag? ¿Do you see any error in the grok match that's preventing the line from being parsed?

Thanks for your help!

2

What do you mean by "it stopped working"?

3

That I managed to make it parse my log, went home and found grokparsefailure in the morning.
And then I could never make it work again.

Pretty sure it was this exact same configuration minus the mutate part.

4

You will always have _grokparsefailure with that configuration because the peer message grok fails on block messages, and the block message grok fails on peer messages.

5

Is there a way to avoid the _grokparsefailure?

6

Check to see if the line is likely to match before executing the grok

if [message] =~ / txs / {
    grok { ... }
} elsif [message] =~ / peers / {
    grok { ... }
}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.