Need to Extract field

Elastic Stack Logstash
1

Hi Experts,

I am using logstash 7.9.3 and i have written the MongoDb connection with Elasticsearch pipeline. Data's are perfectly pushed into Elasticsearch. In that index I need to split one field.

My logstash configuration is:-

input {
mongodb{
uri => 'mongodb://127.0.0.1:27017/testDb'
placeholder_db_dir => 'C:\Data\db'
placeholder_db_name => 'test.db'
collection => 'testmeachine'
batch_size => 10
parse_method => "simple"
}
}
filter{
mutate {
copy => { "_id" => "[@metadata][_id]"}
remove_field => ["_id"]
}
mutate {
copy => { "userId" => "[@metadata][userId]"}
remove_field => ["userId"]
}
mutate {
copy => { "formId" => "[@metadata][formId]"}
remove_field => ["formId"]
}
}
output {
elasticsearch {
action => "index"
index => "mongo01test"
hosts => "http://localhost:9200"
}
stdout{
codec => rubydebug
}
}

Here is the sample field need to change:-

"afterValue" : """{"orderno"=>104, "amount"=>12031, "Save and Close"=>false, "Approved"=>false, "Rejected"=>true}""",

Thanks and Advance,
Sriguruvel.

2

HI experts,

Need help.

Thanks ,
Sriguruvel.

3

If you want some advice, you need to elaborate what change you want.

4

Hi @Tomo_M ,

This is my logstash.conf file:-

image
image835×654 13.9 KB

Here is my Mongo DB data

image
image580×531 13.8 KB

Here is the output of logstash:-

image
image1221×404 17 KB

Here is my Elasticsearch Output:-

image
image815×581 38.5 KB

Some of my data are pushed look like a message field . for example there is a beforeValue field , i need that contant as an array.

Thanks in advance,
Sriguruvel.

5

Please not paste screenshots but use </> to preformat texts. Texts could copy and seachable.

I'm not familiar with MongoDB input plugin, but is there no option about parsing object fields? Though I'm not sure what you exactly mean by "split" the field, JSON filter plugin can parse JSON string field to some object or array.

6

Hi @Tomo_M ,

sorry for that .
I will try JSON filter plugin.

Thanks,
Sriguruvel.

7

And also please check the mongoDB input plugin's document.

8

I suspect this is your problem. Go read the source and see if flatten or dig would work better for your use case.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.