Field list

No. It is not that easy. I need the field in next mutate for further splitting.

see this , you'll understand the intent.

mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"} // No need to send this to Output
       add_field => {"part2" =>"%{[message][1]}"} // No need to send this to Output    
}


mutate {
       split => ["part2","#"]
       add_field => {"part2_1" =>"%{[part2][0]}"} // No need to send this to Output
       add_field => {"part2_2" =>"%{[part2][1]}"} // No need to send this to Output
}


mutate {
       split => ["part2_2","="]
       add_field => {"X" =>"%{[part2_2][0]}"} // This is required in output
       add_field => {"Y" =>"%{[part2_2][1]}"} // This is required in output
}

tell me what change I should do here so that only X , Y goes to output

It is unclear what you want, but you might find it useful to add fields inside [@metadata]. Those are attached to the event, but are not added to the document by the output.

Yes. I tried that but it gives error.
Please see this
https://pastebin.com/65WNbSck

Is it wrong syntactically ?

split => [[@metadata][qravsmanual],"="]

This should be

split => { "[@metadata][qravsmanual]" => "=" }

Edited to fix syntax...

I guess you missed "}" here . Is it a typo ?
split => { "[@metadata][qravsmanual]" => "="] }

here is the latest config after correction

https://pastebin.com/6Pufu6pF

Is this syntactically okay now ?

You can just run logstash from command line with --config.test_and_exit to check syntax.

If you are comfortable using docker containers on your own you can use this tool Web-UI for Logstash filter development

Either way you can test it and if it throws an error, paste here all the details:

  • log contents used as input
  • logstash configuration
  • errors, desired or unexpected output

In fact, if you paste here a sample log line it would be really useful. I presume that, instead of splitting a message in two by a word separator, splitting the result in two again by another word, splitting again by other separators... is a contrieved way to extract the desired information. Maybe grok or kv filters are a simpler solution for your use case.

No, remove the ] from line 19.

when I run this
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

I get this error
https://pastebin.com/2P8V6CDR

Could you please tell , which syntax I am making wrong ?

Here is my logstash.conf
https://pastebin.com/XNpeGme7

@Badger I did that. But there is still syntax error.

Here is the latest logstash.conf ... I have appended the @metadata snippet at the end block.

latest config file
https://pastebin.com/XNpeGme7

What I am missing ?

Hi,
end of the line 135 }} should be }"} ?

@grumo35

corrected this part.

when I run this now
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

This is still giving syntax error

https://pastebin.com/4www1DHK

Here is the latest config file
https://pastebin.com/gtv8RrBS

what I am missing ?

Line 145
split => [[@metadata][request_id],"="] should be ["[ at the begging

I'm not sure of your strange syntax.

You should go for :

split => [ "[FIELDNAME]" , "=" ]

I just change this to

split => { "[@metadata][request_id]" => "=" }

This seems working .

I run again
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

Result :
runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

@grumo35

could you please tell how you are identifying line no of issue ? I dont see that in the error message.

The error message contains the line number

Reason: Expected one of #, ", ', -, [, {, ] at line 145, column 19

Thanks....that helped a lot.

Now I'm getting the fields successfully in the output and also in Kibana.

Kibana screen

But I do not see these fields in the filter (arrow marked) in Kibana.

Is there anything I require to do so that these fields are visible in filter ?

This is because , I want to apply a filter condition for these fields.

That's a kibana question. I do not run kibana. You might want to ask in the kibana forum.

try refreshing the mapping, and make sure the field are "searchable"

refreshing mapping worked !

It helped a lot.

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.