How to convert data types?

Elastic Stack Logstash
1

Hi,
I want to parse the following data which is in the .txt file (separated by tabs):
1 apple tree-a city1
2 banana tree-b city2

input {
  file {
    path => "path-to-file"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{NUMBER:id}\t%{WORD:fruit} etc... }    
  }
}

My input is a txt file and the filter works. My question is, can I use the following (and how):

filter {
  mutate {
    convert => ["id","integer"]
    convert => ["fruit","string"]
    convert => ["tree","string"]
    convert => ["city","string"]
}
}

Because what I get in Kibana is the entire unparsed message with all the fields inside. Logstash says config is fine and runs OK. But something is wrong during mapping data types, as I understand it. Not sure how it should be done otherwise. Thanks for any advise or idea. I'm using ELK 7.

2

You are saying that the grok filter parses [message] into those fields? Because I would not expect that unless you have set config.support_escapes. \t is not parsed as a tab in grok, use a literal tab in the grok pattern (obviously if you use an editor like vi that means you cannot have the expandtab option enabled)

filter {
    mutate {
        convert => ["id","integer"]
        convert => ["fruit","string"]
        convert => ["tree","string"]
        convert => ["city","string"]
    }
}

Not sure if this works or not (logstash can be surprisingly forgiving of using arrays where hashes are expected, but surprisingly unforgiving where duplicate options occur). I would write this as

mutate {
    convert => {
        "id" =>"integer"
        "fruit" => "string"
        "tree" => "string"
        "city" => "string"
    }
}

That said, grok will produce a string by default for any pattern match, and you can adjust that by changing %{NUMBER:id} to %{NUMBER:id:int} and then remove the mutate filter.

3

Thanks for your reply Badger, unfortunately I didn't understand how tabulation works in grok.
Simple example:

1 apple sweet-01
2 lime bitter-02

separated by space

filter {
  grok {
    match => { "message" => "%{NUMBER:id} %{WORD:fruit} (?<taste>[\w+-]+)" }
  }
}

This pattern passes in Kibana grok debugger.
This time my question is, if I add a tab instead of space in my simple data, how grok filter changes, to reflect tabulation ? thanks

4

If you have a tab in your data, you need a tab in your grok pattern.

If you have a space in your data, you need a space in your grok pattern.

Or you could use \s, or perhaps \s+ to match one or more of either.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.