Extract fields from JSON

Elastic Stack Logstash
1

I have this below JSON coming from RabbitMQ

event": {
"payloadContext": {
"messageProfile": {
"domain": "ERP",
"process": "process-01",
"serviceName": "service-01",
"serviceVersion": "4.0.0.RELEASE"
},
"applicationProfile": {
"appName": "app-0",
"appUser": "test"
},
"transactionProfile": {
"transactionDateTime": {
"value": 1472146765000,
"timeZoneCode": null,
"daylightSavingTimeIndicator": null
},
"globalTransactionID": "bb4e273b-c0b6-1378-b2d0-8328971f19d5",
"repostFlag": null,
"transactionMode": null,
"environment": "Test",
"event": null
},
"userArea": null
}
"emailParams": {
"fromAddress": "xxx@abc.com",
"toAddress": "yyy@abc.com",
"subject": xxxxxxxxx,
"template": "xxxxx",
"avoidDuplicate": true,
"attachmentRequired": true,
"ttl": 3600000
},
"ticketParams": null,
"rule": null,
"@version": "1",
"@timestamp": "2016-08-25T17:39:25.442Z"
}

i am looking to extract below fields only and output into elasticsearch

payloadContext/serviceName
applicationProfile/appName
transactionProfile/transactionMode
emailParams/fromAddress
emailParams/toAddress

any input would be very helpful

2

What have you tried?

3

Use the mutate filter to copy/move the fields you want to keep into new fields (presumably you want them at the top level of the event rather than as nested fields) then use the prune filter to delete everything but those fields. Or, if the whole message is nested under a single top-level event field, you can just delete that top-level field after you've saved the fields you're interested in.

1 Like
5

tried below but didn't help,

filter
{
mutate
{
remove_field =>[ "[event][applicationProfile][appName]" ]
}
}

6

We need more details. What did the event look like? What's your configuration? What did you get? What did you expected to get instead?

7

please find below config & event input

  1. config

input{
rabbitmq{
host => "xxxx"
port => "5672"
user => "xxx"
password => "xxx"
exchange => "xxx.EXG"
queue => "xxx.Q"
durable => "true"
vhost => "vhost"
threads => 5
}
}
filter
{
mutate
{
remove_field =>[ "[event][applicationProfile][appName]" ]
}
}

output {
elasticsearch{
hosts => ["0.0.0.0"]
index => "emlticks"
document_type => "emlticks"
}
}

  1. event data

{
"event": {
"payloadContext": {
"messageProfile": {
"domain": "ERP",
"process": "process-a",
"serviceName": "testservice",
"serviceVersion": "1.0"
},
"applicationProfile": {
"appName": "testapp",
"appUser": "user"
},
"transactionProfile": {
"transactionDateTime": {
"value": 1472473590000,
"timeZoneCode": null,
"daylightSavingTimeIndicator": null
},
"globalTransactionID": "61ddb532-8d84-87f1-8cac-dec421523ea0",
"repostFlag": null,
"transactionMode": null,
"environment": "Test",
"event": null
},
"userArea": null
},
"document": {
"eventActivity": {
"event": "PROCESS",
"eventCode": "null",
"eventSubCode": null,
"step": {
"value": null,
"languageID": null
},
"status": {
"value": "FAILED",
"languageID": null
},
"summary": null,
"detail": {
"value": null,
"languageID": null
},
"payload": null
"businessIdentifier": "50063000002VDygAAG",
"alternateBusinessIdentifier": "04kj00000008OS1AAM",
"hostName": "aics360-qas_1",
"threadID": "check.release.task.executor-1"
}
}
},
"emailParams": {
"fromAddress": "xxx@xxx.com",
"toAddress": "xxx@xxx.com",
"subject": "Test- Attention required for service",
"template": "common-email-template",
"avoidDuplicate": true,
"attachmentRequired": true,
"ttl": 3600000
},
"ticketParams": null,
"rule": null,
"@version": "1",
"@timestamp": "2016-08-29T12:26:31.364Z"
}
}

from the event i am looking to extra few properties like payloadContext.domain, payloadContext.process, payloadContext.serviceName, emailParams.fromAddress,emailParams.toAddress, applicationProfile.appName etc.,

8

There is no [event][applicationProfile][appName] field. It's named [event][payloadContext][applicationProfile][appName].

9

yeah checking.... if i want to remove the tag applicationProfile should i do like below ?

filter
{
mutate
{
remove_tag =>[ "[event][payloadContext][applicationProfile]" ]
}
}

10

Keep using remove_field.

Why don't you try it?

11

yeah it worked with remove_field, but tried remove_tag which didn't help