JSON input field

Mbt_Mbt · January 29, 2020, 1:07pm

Good afternoon, I am trying to save the information contained in a field in independent fields for later analysis in kibana.

I have tried actions of the json target, source plugin ... but I don't get any positive results.

The format is as follows,

Can somebody help me?

inhinyera16 · January 29, 2020, 2:38pm

I'm not sure but if you just want to resolve this, you can run another json filter then use the event.EventData.CustomDataCollection.Rows column as source.

Mbt_Mbt · January 29, 2020, 2:59pm

could you give more information please?

Thanks for your time

inhinyera16 · January 29, 2020, 3:13pm

I think, it didnt parse the inner json due to it's an array. I suppose you are using logstash as u mentioned json plugin. Anyways, you can add either of these in the filter blocks.

 json {
 	source => "event.EventData.CustomDataCollection.Rows"
 }

or

 mutate {
 	add_field => { "phone" => "%{[event.EventData.CustomDataCollection.Rows][Phone][Value]}" }
	add_field => { "name" => "%{[event.EventData.CustomDataCollection.Rows][Name][Value]}" }
	add_field => { "lastname" => "%{[event.EventData.CustomDataCollection.Rows][LastName][Value]}" }
}

Then you can remove the event.EventData.CustomDataCollection.Rows after.

Mbt_Mbt · January 29, 2020, 4:55pm

Thank you very much for your time inhinyera16 but it has not worked.

Any idea how I can pass the information to independent fields?

grumo35 · January 29, 2020, 6:36pm

source => "[event][EventData][CustomDataCollections][Rows]"

Mbt_Mbt · January 29, 2020, 10:24pm

Thank you very much for your help and time but I have not been able to extract the fields using the advice received.

Any other idea?

Many thanks

Badger · January 30, 2020, 1:15am

Add

output { stdout { codec => rubydebug } }

and show us what an event looks like. That contains more information than the JSON tab in Kibana.

Mbt_Mbt · January 30, 2020, 7:51am

Hello Badger,

Thanks for your healp,

{
"_index": "bplogs",
"_type": "_doc",
"_id": "dx1t9W8BQpprR2SqZxxM",
"_version": 1,
"_score": null,
"_source": {
"headers": {
"http_version": "HTTP/1.1",
"http_accept": null,
"accept_encoding": "gzip,deflate",
"connection": "Keep-Alive",
"http_host": "localhost:5044",
"http_user_agent": "Manticore 0.6.4",
"content_type": "application/json",
"request_path": "/",
"request_method": "PUT",
"content_length": "1010"
},
"eventdata": "{"EventData":{"CustomDataCollection":{"Rows":[{"Name":{"Value":"Robin"},"LastName":{"Value":"Hood"},"Phone":{"Value":"967778875"}}]},"SessionNumber":15,"StageID":"180fabc4-fbc8-422d-a5d0-e0270f4e42b1","StageName":"Data Gateways::Send Custom Data","StageType":2,"StartDate":"2020-01-30T08:49:43.4891218+01:00","ProcessName":"DataGateway","PageName":"Main Page","ObjectName":"Data Gateways","actionName":"Send Custom Data"},"EventType":3}",
"@version": "1",
"@timestamp": "2020-01-30T07:49:45.118Z",
"event": {
"EventType": 3,
"EventData": {
"CustomDataCollection": {
"Rows": [
{
"Name": {
"Value": "Robin"
},
"Phone": {
"Value": "967778875"
},
"LastName": {
"Value": "Hood"
}
}
]
},
"SessionNumber": 15,
"StageType": 2,
"ProcessName": "DataGateway",
"StageName": "Data Gateways::Send Custom Data",
"PageName": "Main Page",
"ObjectName": "Data Gateways",
"StartDate": "2020-01-30T08:49:43.4891218+01:00",
"StageID": "180fabc4-fbc8-422d-a5d0-e0270f4e42b1",
"actionName": "Send Custom Data"
}
}
},
"fields": {
"event.EventData.StartDate": [
"2020-01-30T07:49:43.489Z"
],
"@timestamp": [
"2020-01-30T07:49:45.118Z"
]
},
"sort": [
1580370585118
]
}{
"_index": "bplogs",
"_type": "_doc",
"_id": "dx1t9W8BQpprR2SqZxxM",
"_version": 1,
"_score": null,
"_source": {
"headers": {
"http_version": "HTTP/1.1",
"http_accept": null,
"accept_encoding": "gzip,deflate",
"connection": "Keep-Alive",
"http_host": "localhost:5044",
"http_user_agent": "Manticore 0.6.4",
"content_type": "application/json",
"request_path": "/",
"request_method": "PUT",
"content_length": "1010"
},
"eventdata": "{"EventData":{"CustomDataCollection":{"Rows":[{"Name":{"Value":"Robin"},"LastName":{"Value":"Hood"},"Phone":{"Value":"967778875"}}]},"SessionNumber":15,"StageID":"180fabc4-fbc8-422d-a5d0-e0270f4e42b1","StageName":"Data Gateways::Send Custom Data","StageType":2,"StartDate":"2020-01-30T08:49:43.4891218+01:00","ProcessName":"DataGateway","PageName":"Main Page","ObjectName":"Data Gateways","actionName":"Send Custom Data"},"EventType":3}",
"@version": "1",
"@timestamp": "2020-01-30T07:49:45.118Z",
"event": {
"EventType": 3,
"EventData": {
"CustomDataCollection": {
"Rows": [
{
"Name": {
"Value": "Robin"
},
"Phone": {
"Value": "967778875"
},
"LastName": {
"Value": "Hood"
}
}
]
},
"SessionNumber": 15,
"StageType": 2,
"ProcessName": "DataGateway",
"StageName": "Data Gateways::Send Custom Data",
"PageName": "Main Page",
"ObjectName": "Data Gateways",
"StartDate": "2020-01-30T08:49:43.4891218+01:00",
"StageID": "180fabc4-fbc8-422d-a5d0-e0270f4e42b1",
"actionName": "Send Custom Data"
}
}
},
"fields": {
"event.EventData.StartDate": [
"2020-01-30T07:49:43.489Z"
],
"@timestamp": [
"2020-01-30T07:49:45.118Z"
]
},
"sort": [
1580370585118
]
}

Thanks

Mbt_Mbt · January 30, 2020, 9:44am

tried adding this code

ruby {
code => '
event.get("[event][EventData][CustomDataCollection][Rows]").each { |x|
k = x
v = x["Value"]
if [ "Phone", "Name","LastName"].include? k
event.set(k, v)
end
}'
}

but i doesnt work

Mbt_Mbt · January 30, 2020, 10:43am

tried

json{
source => "eventdata"
target => "parsedJson"
remove_field=>["eventdata"]
}

mutate {
add_field => {
"firstname" => "%{[parsedJson][EventData][CustomDataCollection][Rows][Name]}"
}
}

too and still not getting the expected results

thanks

inhinyera16 · January 30, 2020, 11:21am

Ok so here you go, this works on my end

input {
	file {
		path => "/tmpBIS/ror/elastic_env/logstash-6.7.1/log"
		start_position => "beginning"
 		sincedb_path => "/dev/null"
	}
}


filter {

	json {
		source => "message"
	}

	mutate {
		add_field => { "[EventData][CustomDataCollection_Name]" => "%{[EventData][CustomDataCollection][Rows][0][Name][Value]}" }
		add_field => { "[EventData][CustomDataCollection_LastName]" => "%{[EventData][CustomDataCollection][Rows][0][LastName][Value]}" }
		add_field => { "[EventData][CustomDataCollection_Phone]" => "%{[EventData][CustomDataCollection][Rows][0][Phone][Value]}" }
	}

	mutate {
		remove_field => [ "message","host","@version","path","@timestamp","[EventData][CustomDataCollection]" ]
	}
}
output {
	stdout {codec=> rubydebug}
}

Mbt_Mbt · January 30, 2020, 12:52pm

inhinyera16,

Thank you very much for your help,

It's a pleasure that there are people like you.

inhinyera16 · January 30, 2020, 1:23pm

thanks @Mbt_Mbt!

Ps: Please mark as "Solution" if you think this helps

system · February 27, 2020, 1:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.