Logstash split fields into new fields(field1, field2, field3, field4, field5)


(Mokete Mokoena) #1

Hi

I am importing data from mssql database into Elasticsearch.

I am trying to split field "my_message" that is structured as follows:

"{1:Mokete}{2:Mokoena}{3:{888:Logstash}}{4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}"

I would like to split the above field to :
field1: {1:Name}
field2: {2:Surname}
field3: {3:{888:Logstash}}
field4: {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }
field5: {5:{MY:00000000}{ZAR:Testing}{Test:}}

I tried the following filter:

filter {
mutate
{
split => { "my_message" => "{?:"}
add_field =>
{
"field1" => "%{[my_message][1]}"
"field2" => "%{[my_message][2]}"
"field3" => "%{[my_message][3]}"
"field4" => "%{[my_message][4]}"
"field5" => "%{[my_message][5]}"
}
}
}

this gives me the following results:
field1: {1:Name} {2:Surname} {3:{888:Logstash}} {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}

Expectation:
to go throught "my_message" field and where it finds a "{1:", "{2:","{3:","{4:","{5:" split the message and return :
field1: Name}
field2: Surname}
field3: {888:Logstash}}
field4: \n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }
field5: {MY:00000000}{ZAR:Testing}{Test:}}

Thanks,
Mokete


(Magnus Bäck) #2

I'd recommend a grok or a dissect filter for this problem.


(Mokete Mokoena) #3

Hi @magnusbaeck

Thanks for the quick reply.

Attempted grok, dissect and kv filter but couldn't get the syntax correct as they we all throwing errors.

Pipeline aborted due to error {:exception=>"Grok::PatternError", :error=>"pattern %{my_message} not defined", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:123:in compile'", "org/jruby/RubyKernel.java:1479:inloop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.4/lib/grok-pure.rb:93:in compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:inregister'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:inregister'", "org/jruby/RubyHash.java:1342:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:inregister'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:182:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:182:in start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}

Please advice on the syntax for both grok and dissect.

Thanks


(Magnus Bäck) #4

And what configuration gave you the error above?


(Mokete Mokoena) #5

@magnusbaeck

I used the below :

filter {
grok {
match => { "message" => "%{my_message}" }
}
}


(Magnus Bäck) #6

That looks like dissect syntax in a grok filter. Switch to a dissect filter or use the grok constructor web site to construct a grok expression that you can use.


(Mokete Mokoena) #7

@magnusbaeck

swithed to disssect filter and getting the following

fetched an invalid config {:config=>************* :reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: no such file to load -- logstash/filters/dissect", :level=>:error}


(Magnus Bäck) #8

What version of Logstash?


(Mokete Mokoena) #9

on grok website

provided input:
{1:Name} {2:Surname} {3:{888:Logstash}} {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}

pattern :
(?{\d:.}) (?{\d:.})(?{\d:.*})

results:

{
"message": [
[
"{1:Name} {2:Surname} {3:{888:Logstash}}"
]
]
}

pattern :
(?{\d:.}) (?{\d:.})(?{\d:.})(?{\d:.})

results:
No Matches


(Mokete Mokoena) #10

@magnusbaeck

version 2.4


(Magnus Bäck) #11

2.4 is probably too old for dissect. You might be able to install the plugin via logstash-plugin install but otherwise you have to upgrade Logstash.


(Mokete Mokoena) #12

@magnusbaeck, i managed to construct grok syntax from grok constructor web site but when i run Logstash i get the following output.

"_index": "messages",
"_type": "message",
"_id": "1111",
"_score": 1,
"_source": {
"my_message": "{1:Name} {2:Surname} {3:{888:Logstash}} {4:\n:20:Testing\n:LG:Logstash\n:ES:Elasticsearch,\n:CURR:ZAR,\n:HM:/Home\nHOME Jake\n111 FIRST ROAD \n ZA }{5:{MY:00000000}{ZAR:Testing}{Test:}}",
"tags": [
"_grokparsefailure"
]
}
}

Thanks


(Magnus Bäck) #13

To debug, start with the simplest possible grok expression. Does it work? If yes, continue by adding more and more and it breaks. That narrows down the problem.


(Mokete Mokoena) #14

This is my expression:

filter {
grok {
match => { "%{my_message}" => "(?{1:.*?})" }
}
}


(Magnus Bäck) #15

Replace %{fin_format} with message since that's the field you want to parse.

Always format configuration snippets as preformatted text so the posting software doesn't mangle it.


(system) #17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.