Need to Extract field

[SRIGURU_VEL]

Hi Experts,

I am using logstash 7.9.3 and i have written the MongoDb connection with Elasticsearch pipeline. Data's are perfectly pushed into Elasticsearch. In that index I need to split one field.

My logstash configuration is:-

input {
mongodb{
uri => 'mongodb://127.0.0.1:27017/testDb'
placeholder_db_dir => 'C:\Data\db'
placeholder_db_name => 'test.db'
collection => 'testmeachine'
batch_size => 10
parse_method => "simple"
}
}
filter{
mutate {
copy => { "_id" => "[@metadata][_id]"}
remove_field => ["_id"]
}
mutate {
copy => { "userId" => "[@metadata][userId]"}
remove_field => ["userId"]
}
mutate {
copy => { "formId" => "[@metadata][formId]"}
remove_field => ["formId"]
}
}
output {
elasticsearch {
action => "index"
index => "mongo01test"
hosts => "http://localhost:9200"
}
stdout{
codec => rubydebug
}
}

Here is the sample field need to change:-

"afterValue" : """{"orderno"=>104, "amount"=>12031, "Save and Close"=>false, "Approved"=>false, "Rejected"=>true}""",

Thanks and Advance,
Sriguruvel.

[SRIGURU_VEL]

HI experts,

Need help.

Thanks ,
Sriguruvel.

[Tomo_M (Tomohiro Mitani)]

@SRIGURU_VEL

Here is the sample field need to change:-

"afterValue" : """{"orderno"=>104, "amount"=>12031, "Save and Close"=>false, "Approved"=>false, "Rejected"=>true}""",

If you want some advice, you need to elaborate what change you want.

[SRIGURU_VEL]

Hi @Tomo_M ,

This is my logstash.conf file:-

Here is my Mongo DB data

Here is the output of logstash:-

Here is my Elasticsearch Output:-

Some of my data are pushed look like a message field . for example there is a beforeValue field , i need that contant as an array.

Thanks in advance,
Sriguruvel.

[Tomo_M (Tomohiro Mitani)]

Please not paste screenshots but use </> to preformat texts. Texts could copy and seachable.

I'm not familiar with MongoDB input plugin, but is there no option about parsing object fields? Though I'm not sure what you exactly mean by "split" the field, JSON filter plugin can parse JSON string field to some object or array.

[SRIGURU_VEL]

Hi @Tomo_M ,

sorry for that .
I will try JSON filter plugin.

Thanks,
Sriguruvel.

[Tomo_M (Tomohiro Mitani)]

@Tomo_M

is there no option about parsing object fields?

And also please check the mongoDB input plugin's document.

[Badger]

@SRIGURU_VEL

parse_method => "simple"

I suspect this is your problem. Go read the source and see if flatten or dig would work better for your use case.