Logstash grok pattern for ping

[geokal (Giorgos Kal)]

Hi everybody, have this sample from a log:

   Tue Mar 27 06:51:48 2018 PING www.google.com (172.217.169.100) 56(84) bytes of data.
    64 bytes from sof02s31-in-f4.1e100.net (172.217.169.100): icmp_seq=1 ttl=128 time=17.4 ms

    --- www.google.com ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 17.482/17.482/17.482/0.000 ms

I want to make a grok pattern for logstash and extract things like TIMESTAMP IPV4 TTL , as well as the RTT values min/avg/max from last 2 lines. This log is from a pinging script to the same IP every second or so. I guess I need a multiline pattern to take the values for each of those 6 lines at once? Any Help would be great!!!Thanks

[Badger]

    codec => multiline { 
      pattern => "^(Sun|Mon|Tue|Wed|Thu|Fri|Sat)" negate => true what => "previous"
    }

might do it.

[geokal (Giorgos Kal)]

Thanks a lot @Badger for this , any help for the fist part like the first 2 lines ?

[Badger]

Exactly which fields do you want to extract?

[geokal (Giorgos Kal)]

Fields to extract

So far I got these :
%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} PING %{HOSTNAME}

[Badger]

That link does not work for me. Anyways, I would do it using this. Since you only do one ping min == avg == max and mdev == 0, so just pull one number out.

  dissect { mapping => { "message" => "%{ts} %{+ts} %{+ts} %{+ts} %{+ts} PING %{target} (%{ip})%{}" } }
  grok { match => { "message" => "min/avg/max/mdev = %{NUMBER}/%{NUMBER:avg:float}/%{NUMBER}/%{NUMBER} ms" } }

[atira (Attila Boncok)]

You're not getting any fields like that. Also you might want to get the timestamp in one field.
I recommend checking the Grok patterns and the Grok debugger.
Grok patterns
Grok debugger

You can use custom patterns, read about them here: click to guide
I recommend using the custom pattern file.

Example custom pattern file:
CUSTOMDATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
And used in the actual logstash config:
%{CUSTOMDATE:timestamp}

Try it out in the Grok debugger

Then use it in the date filter plugin:
date { match => [ "timestamp" , "EEE MMM dd HH:mm:ss YYYY" ] }

[atira (Attila Boncok)]

A remark for the multiline part, check this.

If you are using a Logstash input plugin that supports multiple hosts, such as the beats input plugin, you should not use the multiline codec to handle multiline events.

They rather recommend handling multilines with filebeat. According to your sample, every continued line starts with a space, so
multiline.pattern: ^\s
multiline.negate: false
multiline.match: after

[atira (Attila Boncok)]

Here you go, with the custom pattern posted above:

%{CUSTOMDATE:timestamp} %{WORD} %{HOSTNAME:hostname} \(%{IP:ipv4}\) %{DATA} ttl=%{INT:ttl} %{DATA} rtt min/avg/max/mdev = %{NUMBER:rtt_min}/%{NUMBER:rtt_avg}/%{NUMBER:rtt_max}/%{NUMBER:rtt_mdev} ms

Check it out in the Grok debugger.
Be sure to create the appropriate mapping in ES.

edit: revisiting after taking a look at dissect, Badger's filters might be better performance-wise, though they don't include TTL.

[geokal (Giorgos Kal)]

Thanks, guys!!! Literally, save my a#@!!! Currently working on a mininet project and getting around elk was pretty hard especially configure things between VM's . Grok debugger is where i test stuff.

[geokal (Giorgos Kal)]

Sorry, these were the fields i was reffering, but you guessed right without the photo!

[geokal (Giorgos Kal)]

Hi @atira and @Badger, after a nightmare setting up right the filebeat to send logs and get indices to kibana, I got a json response from kibana:

  {
  "_index": "filebeat-2018.04.03",
  "_type": "doc",
  "_id": "_VR7i2IB9grcFnMkJeXz",
  "_version": 1,
  "_score": null,
  "_source": {
    "offset": 44205,
    "ip": "(172.217.17.174",
    "tags": [
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "source": "/home/mininet/mininet/examples/ping_v1e+00_google.com_2018-04-03_4h29m19s.log",
    "target": "",
    "ts": "64 bytes from sof02s21-in-f14.1e100.net ",
    "message": "64 bytes from sof02s21-in-f14.1e100.net (172.217.17.174): icmp_seq=1 ttl=128 time=17.2 ms",
    "@timestamp": "2018-04-03T11:32:05.319Z",
    "prospector": {
      "type": "log"
    },
    "host": "mininet-vm",
    "beat": {
      "name": "mininet-vm",
      "hostname": "mininet-vm",
      "version": "6.2.3"
    },
    "@version": "1"
  },
  "fields": {
    "@timestamp": [
      "2018-04-03T11:32:05.319Z"
    ]
  },
  "sort": [
    1522755125319
  ]
}

so I changed the grok pattern to this:

min/avg/max/mdev = %{NUMBER}/%{NUMBER:float:avg}/%{NUMBER}/%{NUMBER} ms
just a switch form avg: float to ==> float:avg, I guess I must finetune the grok to make it right 100% and get the avg RTT and would probably need additional help, also the dissect field must be present with the grok right?Thanks all for your kind help!! Greetings from Greece!!