Logstash split fields into new fields(field1, field2, field3, field4, field5)

[moketemokoena]

Hi

I am importing data from mssql database into Elasticsearch.

I am trying to split field "my_message" that is structured as follows:

"{1:Mokete}{2:Mokoena}{3:{888:Logstash}}{4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}"

I would like to split the above field to :
field1: {1:Name}
field2: {2:Surname}
field3: {3:{888:Logstash}}
field4: {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }
field5: {5:{MY:00000000}{ZAR:Testing}{Test:}}

I tried the following filter:

filter {
mutate
{
split => { "my_message" => "{?:"}
add_field =>
{
"field1" => "%{[my_message][1]}"
"field2" => "%{[my_message][2]}"
"field3" => "%{[my_message][3]}"
"field4" => "%{[my_message][4]}"
"field5" => "%{[my_message][5]}"
}
}
}

this gives me the following results:
field1: {1:Name} {2:Surname} {3:{888:Logstash}} {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}

Expectation:
to go throught "my_message" field and where it finds a "{1:", "{2:","{3:","{4:","{5:" split the message and return :
field1: Name}
field2: Surname}
field3: {888:Logstash}}
field4: \n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }
field5: {MY:00000000}{ZAR:Testing}{Test:}}

Thanks,
Mokete

[magnusbaeck (Magnus Bäck)]

I'd recommend a grok or a dissect filter for this problem.

[moketemokoena]

Hi @magnusbaeck

Thanks for the quick reply.

Attempted grok, dissect and kv filter but couldn't get the syntax correct as they we all throwing errors.

Pipeline aborted due to error {:exception=>"Grok::PatternError", :error=>"pattern %{my_message} not defined", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in compile'", "org/jruby/RubyKernel.java:1479:inloop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:inregister'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:inregister'", "org/jruby/RubyHash.java:1342:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:182:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:182:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}

Please advice on the syntax for both grok and dissect.

Thanks

[magnusbaeck (Magnus Bäck)]

And what configuration gave you the error above?

[moketemokoena]

@magnusbaeck

I used the below :

filter {
grok {
match => { "message" => "%{my_message}" }
}
}

[magnusbaeck (Magnus Bäck)]

That looks like dissect syntax in a grok filter. Switch to a dissect filter or use the grok constructor web site to construct a grok expression that you can use.

[moketemokoena]

@magnusbaeck

swithed to disssect filter and getting the following

fetched an invalid config {:config=>************* :reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: no such file to load -- logstash/filters/dissect", :level=>:error}

[magnusbaeck (Magnus Bäck)]

What version of Logstash?

[moketemokoena]

on grok website

provided input:
{1:Name} {2:Surname} {3:{888:Logstash}} {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}

pattern :
(?{\d:.}) (?{\d:.})(?{\d:.*})

results:

{
"message": [
[
"{1:Name} {2:Surname} {3:{888:Logstash}}"
]
]
}

pattern :
(?{\d:.}) (?{\d:.})(?{\d:.})(?{\d:.})

results:
No Matches

[moketemokoena]

@magnusbaeck

version 2.4

[magnusbaeck (Magnus Bäck)]

2.4 is probably too old for dissect. You might be able to install the plugin via logstash-plugin install but otherwise you have to upgrade Logstash.

[moketemokoena]

@magnusbaeck, i managed to construct grok syntax from grok constructor web site but when i run Logstash i get the following output.

"_index": "messages",
"_type": "message",
"_id": "1111",
"_score": 1,
"_source": {
"my_message": "{1:Name} {2:Surname} {3:{888:Logstash}} {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}",
"tags": [
"_grokparsefailure"
]
}
}

Thanks

[magnusbaeck (Magnus Bäck)]

To debug, start with the simplest possible grok expression. Does it work? If yes, continue by adding more and more and it breaks. That narrows down the problem.

[moketemokoena]

@magnusbaeck

To debug, start with the simplest possible grok expression. Does it work? If yes, continue by adding more and more and it breaks. That narrows down the problem.

This is my expression:

filter {
grok {
match => { "%{my_message}" => "(?{1:.*?})" }
}
}

[magnusbaeck (Magnus Bäck)]

Replace %{fin_format} with message since that's the field you want to parse.

Always format configuration snippets as preformatted text so the posting software doesn't mangle it.