Advise for logstash filter

Elastic Stack Logstash
1

Hi,

I am trying to parse below line and due to time stamp and varying delimiter I am unable to use it. Could you please advise?

18/09/2016 11:09:12:AYS2345:2 dcs:0 meta->info:0 total:6 ym->distila:56

I am looking to format as below.

{
Time: 18/09/2016 11:09:12
user: AYS2345
gear:2
dcs:0
meta->info:0
total:6
ym->distila:56
}

2 
grok { match => [ "message", "%{DATESTAMP:Time}:%{USER:user}:%{INT:gear} dcs:%{INT:dcs} meta->info:%{INT:meta} total:%{INT:total} ym->distila:%{INT:ym}" ] }

will produce:

{
  "Time": [
    [
      "18/09/2016 11:09:12"
    ]
  ],
  "user": [
    [
      "AYS2345"
    ]
  ],
  "gear": [
    [
      "2"
    ]
  ],
  "dcs": [
    [
      "0"
    ]
  ],
  "meta": [
    [
      "0"
    ]
  ],
  "total": [
    [
      "6"
    ]
  ],
  "ym": [
    [
      "56"
    ]
  ]
}

You can't have -> in a field name, I think.

You could also do a combination of grok+dissect filter by filtering out Time with grok and put the rest into a field, then use dissect on the rest since those can be easily specified with delimiters.

3

Hi Atira,

Thanks for the response. I am using logstash 5.0 and it seems that spaces between fields is causing an issue. Also, is there a way to test the logstash filter before we apply?

4

You provided only one example, the filter above parsed it successfully.
Do your events have varying whitespaces? It is possible to handle them, but we need more examples to see where these are.

Well, not with Logstash as far as I know.
You can use Grok Debugger to test your patterns against real examples.

5

Hi Atira,

I am able to figure it out with gork debugger. Thanks for the help.

  1. Since the resultant fields are in strings, are they compatible for line chart? or we need to convert it to numbers?
  2. I have multiple filebeats configured in different hosts and each has different format. How can include all in one logstash conf? Here I am shipping all to one logstash host.

Thanks,
Maddy

6

Actually, there is no string type in Elasticsearch. There are two types that handle strings:

  1. text: searchable, but no aggregation possible (that means: no statistics)
  2. keyword: not searchable (you'll have to use exact match), but you can aggregate it
    If you want to work with numbers, you'll have to map them.

Multiple Logstash configuration is possible only with conditionals.
For example, Filebeat sends the host name in the source field (if I remember correctly), so you can do a conditional like:

if [source] = "hostname" { <put your code here> }

If you have a lot of different configs, it might become a conditional hell, but in Logstash 5.x, there is no other way AFAIK.
Logstash 6.x accepts multiple config files.

7

This is the config, I am using.

input {
beats {
port => 5044
}

}

if "totaldist" in [tags] {

  grok { match => [ "message", "%{DATESTAMP:Time}:%{USER:user}:%{INT:gear} dcs:%{INT:dcs} meta->info:%{INT:meta} total:%{INT:total} ym->distila:%{INT:ym}" ] }
  
  mutate {
  remove_field => ["beat.hostname","beat.name","beat.version","input_type","type","offset","@version","_type","_id","_score:" ]
  }
if "_grokparsefailure" in [tags] {
              drop { }
          }
		
}

output {
if [tags] == "totaldist" {
elasticsearch {
hosts => ["mesta:9200"]
index => "totaldist-%{+YYYY.MM.dd}"
}
}
}

8

And your question is...? :slight_smile:

If you're dropping falsely parsed events, you might not realize you don't have all events in ES and even if you realize, you won't know, why the parsing failed.

9

Hi Atira,

The question I have is all the data is going to "message". But I looking to split the fields in "message" section like key value pair. So that I can use the values in line chart.

Also, I see the field names are not coming out as expected.

{
Time: 18/09/2016 11:09:12
user: AYS2345
gear:2
dcs:0
meta->info:0
total:6
ym->distila:56
}

message:05/03/2017 04:13:00:7611:AYS2345 dcs:1 meta->infoc:0 total:65 ym->distilla:53

The end goal is to draw line chart with values extracted from dcs, info, total and ym

Appreciate your help.

10

I fixed it thanks.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.