Field list

Elastic Stack Logstash
1

I have this entry in logstash.conf

mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}      
}


mutate {
       split => ["part2","#"]
       add_field => {"part2_1" =>"%{[part2][0]}"}
       add_field => {"part2_2" =>"%{[part2][1]}"}

}

This adds part1 , part2 , part2_1 and part2_2 fields in Kibana's Available field list.

But my requirement is to add only part2_2 field . rest of the fields are not required in Kibana.

What changes I should make here so that only part2_2 field is added in Kibana's Available field list.

2

If you do not want the other fields then do not add them If you need to remove existing fields then use mutate+remove_field.

3

How ? I dont want field part1

You mean change this

mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}      
}

to this

mutate {
       split => ["message","Employee"]
       part1 =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}      
}

look at part1 . Is this correct syntax not to add the field? I am stuck at this spot.

4

Change

mutate {
   split => ["message","Employee"]
   add_field => {"part1" =>"%{[message][0]}"}
   add_field => {"part2" =>"%{[message][1]}"}      
}

to

mutate {
   split => ["message","Employee"]
   add_field => {"part2" =>"%{[message][1]}"}      
}
5

No. It is not that easy. I need the field in next mutate for further splitting.

see this , you'll understand the intent.

mutate {
       split => ["message","Employee"]
       add_field => {"part1" =>"%{[message][0]}"} // No need to send this to Output
       add_field => {"part2" =>"%{[message][1]}"} // No need to send this to Output    
}


mutate {
       split => ["part2","#"]
       add_field => {"part2_1" =>"%{[part2][0]}"} // No need to send this to Output
       add_field => {"part2_2" =>"%{[part2][1]}"} // No need to send this to Output
}


mutate {
       split => ["part2_2","="]
       add_field => {"X" =>"%{[part2_2][0]}"} // This is required in output
       add_field => {"Y" =>"%{[part2_2][1]}"} // This is required in output
}

tell me what change I should do here so that only X , Y goes to output

6

It is unclear what you want, but you might find it useful to add fields inside [@metadata]. Those are attached to the event, but are not added to the document by the output.

7

Yes. I tried that but it gives error.
Please see this
https://pastebin.com/65WNbSck

Is it wrong syntactically ?

8 
split => [[@metadata][qravsmanual],"="]

This should be

split => { "[@metadata][qravsmanual]" => "=" }

Edited to fix syntax...

9

I guess you missed "}" here . Is it a typo ?
split => { "[@metadata][qravsmanual]" => "="] }

here is the latest config after correction

https://pastebin.com/6Pufu6pF

Is this syntactically okay now ?

10

You can just run logstash from command line with --config.test_and_exit to check syntax.

If you are comfortable using docker containers on your own you can use this tool Web-UI for Logstash filter development

Either way you can test it and if it throws an error, paste here all the details:

  • log contents used as input
  • logstash configuration
  • errors, desired or unexpected output

In fact, if you paste here a sample log line it would be really useful. I presume that, instead of splitting a message in two by a word separator, splitting the result in two again by another word, splitting again by other separators... is a contrieved way to extract the desired information. Maybe grok or kv filters are a simpler solution for your use case.

11

No, remove the ] from line 19.

12

when I run this
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

I get this error

Could you please tell , which syntax I am making wrong ?

Here is my logstash.conf

13

@Badger I did that. But there is still syntax error.

Here is the latest logstash.conf ... I have appended the @metadata snippet at the end block.

latest config file
https://pastebin.com/XNpeGme7

What I am missing ?

14

Hi,
end of the line 135 }} should be }"} ?

15

@grumo35

corrected this part.

when I run this now
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

This is still giving syntax error

Here is the latest config file

what I am missing ?

16

Line 145
split => [[@metadata][request_id],"="] should be ["[ at the begging

I'm not sure of your strange syntax.

You should go for :

split => [ "[FIELDNAME]" , "=" ]

17

I just change this to

split => { "[@metadata][request_id]" => "=" }

This seems working .

I run again
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit

Result :
runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

18

@grumo35

could you please tell how you are identifying line no of issue ? I dont see that in the error message.

19

The error message contains the line number

Reason: Expected one of #, ", ', -, [, {, ] at line 145, column 19
20

Thanks....that helped a lot.

Now I'm getting the fields successfully in the output and also in Kibana.

Kibana screen

But I do not see these fields in the filter (arrow marked) in Kibana.

Is there anything I require to do so that these fields are visible in filter ?

This is because , I want to apply a filter condition for these fields.