Help with logstash json output to split a field in multiple variables and combine them to a new field

micruzz82 · March 5, 2018, 8:33pm

Hi All

I have this JSON output as below.

{ "totalCount": "1", "imdata": [ { "l1PhysIf": { "attributes": { "adminSt": "up", "autoNeg": "on", "brkoutMap": "none", "bw": "0", "childAction": "", "delay": "1", "descr": "srv", "dn": "topology/pod-1/node-403/sys/phys-[eth1/1]", "dot1qEtherType": "0x8100", "ethpmCfgFailedBmp": "", "ethpmCfgFailedTs": "00:00:00:00.000", "ethpmCfgState": "0", "fecMode": "inherit", "id": "eth1/1", "inhBw": "unspecified", "layer": "Layer2", "lcOwn": "local", "linkDebounce": "100", "linkLog": "default", "mdix": "auto", } } } ] }

The field I am interested in is "dn": "topology/pod-1/node-403/sys/phys-[eth1/1]".

and I would like to extract this and combine the values into a new field as "interface" => 403/1/1

And when running a query in elasticsearch for "interface" and "linkDebounce", I should get the output as 100.

Would anyone be able to help me on how to go about this please if you have experience in this.

Thanks very much.

MC

tag_v · March 5, 2018, 8:49pm

to create a new field you can use mutate add_field. but, from where you are getting "403/1/1". hoping 403 from dn and what about 1/1 ? end part of dn or static ?

magnusbaeck · March 5, 2018, 8:54pm

Your example isn't valid JSON but perhaps you have simplified it.

Does the imdata always contain an array with a single element like in this case?

If yes, use a grok filter to parse the [imdata][0][l1PhysIf][attributes][dn] field and extract the interesting pieces. You can then use add_field in the same grok filter or in a separate mutate filter to piece together the different field values that you're picking up from different parts of the string.

micruzz82 · March 5, 2018, 9:12pm

Hi Magnus, yes I did truncate it, I have removed the last comma after auto so that it is a valid JSON.

However that was only one instance of an API query.

Here is another query output for an interface.

{
"totalCount": "1",
"imdata": [
{
"eqptIngrTotal5min": {
"attributes": {
"bytesAvg": "11464546",
"bytesBase": "0",
"bytesCum": "19503139467272",
"bytesLast": "12658946",
"bytesMax": "12933282",
"bytesMin": "9452679",
"bytesPer": "45858186",
"bytesRate": "1146454.650000",
"bytesRateAvg": "1144584.823232",
"bytesRateLast": "1150813.272727",
"bytesRateMax": "1201475.444444",
"bytesRateMin": "1050297.666667",
"bytesRateSpct": "0",
"bytesRateThr": "",
"bytesRateTr": "0.000000",
"bytesRateTrBase": "1147684.594554",
"bytesRateTtl": "4578339.292929",
"bytesSpct": "0",
"bytesThr": "",
"bytesTr": "0",
"bytesTrBase": "346504702",
"childAction": "",
"cnt": "4",
"dn": "topology/pod-1/node-103/sys/phys-[eth1/1]/CDeqptIngrTotal5min",
"lastCollOffset": "40",
"modTs": "never",
"pktsAvg": "10995",
"pktsBase": "0",
"pktsCum": "12823322256",
"pktsLast": "8292",
"pktsMax": "18740",
"pktsMin": "6921",
"pktsPer": "43983",
"pktsRate": "1099.575000",
"pktsRateAvg": "1085.224747",
"pktsRateLast": "753.818182",
"pktsRateMax": "1703.636364",
"pktsRateMin": "753.818182",
"pktsRateSpct": "0",
"pktsRateThr": "",
"pktsRateTr": "0.000000",
"pktsRateTrBase": "795.597602",
"pktsRateTtl": "4340.898990",
"pktsSpct": "0",
"pktsThr": "",
"pktsTr": "0",
"pktsTrBase": "236893",
"repIntvEnd": "2018-03-05T21:10:29.990+00:00",
"repIntvStart": "2018-03-05T21:09:49.990+00:00",
"status": "",
"utilAvg": "0",
"utilLast": "0",
"utilMax": "0",
"utilMin": "0",
"utilSpct": "0",
"utilThr": "",
"utilTr": "0",
"utilTrBase": "0",
"utilTtl": "0"
}
}
}
]
}

@tag_v the 1/1 was part of the dn field which had eth1/1

tag_v · March 5, 2018, 9:32pm

in this scenario as magnus mentioned you need to choose grok patterns to extract node number and eth port of "dn" field. (Assuming imdata array has only 1 dn field)

filter {
if [imdata][0][l1PhysIf][attributes][dn] {
	grok {
		match => { "[imdata][0][l1PhysIf][attributes][dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}\]%{GREEDYDATA}" }
	}
	mutate {
		add_field => { "interface" => "%{node}/%{eth}" }
		remove_field => ["node","eth"]
	}
}
}

micruzz82 · March 5, 2018, 9:39pm

Thank you @tag_v and @magnusbaeck

I have run a few more API queries and yes it imdata only has 1 dn field in it.

I will test your filter and see how we can port them into elasticsearch. This will give us very good information on interface stats and hopefully we can build good visualizations for our monitoring.

Many thanks again team.

micruzz82 · March 9, 2018, 2:32pm

I had another question from the above, I was trying to build a conditional filter for the 3 values as below.

topology/pod-1/node-104/sys/phys-[eth1/6]/CDeqptIngrBytes5min

topology/pod-1/node-101/sys/aggr-[po6]/CDeqptIngrBytes5min

topology/pod-1/node-101/sys/mgmt-[mgmt0]/CDeqptIngrBytes5min

This is my filter.

filter {
if [imdata][0][eqptIngrBytes5min][attributes][dn] {

if [aggr] in "[imdata][0][eqptIngrBytes5min][attributes][dn]" {
grok {
match => { "[imdata][0][eqptIngrBytes5min][attributes][dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}po%{DATA:po}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/po-%{po}" }
remove_field => ["node","po"]
}

}

else if [mgmt] in "[imdata][0][eqptIngrBytes5min][attributes][dn]" {
grok {
match => { "[imdata][0][eqptIngrBytes5min][attributes][dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}mgmt%{DATA:mgmt}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/mgmt-%{mgmt}" }
remove_field => ["node","mgmt"]
}

}

else {
grok {
match => { "[imdata][0][eqptIngrBytes5min][attributes][dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/%{eth}" }
remove_field => ["node","eth"]
}

}

}
}

However it doesn't seem to be working.

Can you advise what I need to change from the filter so that the 3 variables can match accordingly and set the interface to the appropriate value.

Thanks for your help

Regards

MC

micruzz82 · March 9, 2018, 3:14pm

I figured it out in the end for those it may help.

filter {
if [dn] {

if "aggr" in [dn] {
grok {
match => { "[dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}po%{DATA:po}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/portchannel-%{po}" }
remove_field => ["node","po"]
}

}

else if "mgmt" in [dn] {
grok {
match => { "[dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}mgmt%{DATA:mgmt}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/mgmt-%{mgmt}" }
remove_field => ["node","mgmt"]
}

}

else {
grok {
match => { "[dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/%{eth}" }
remove_field => ["node","eth"]
}

}

}
}

Thanks for @tag_v and @magnusbaeck for the initial guidance on this.

micruzz82 · March 9, 2018, 4:40pm

I just had one more query to find the correct way to add the data type.

Right now I see a ? in regards to the type.

Could you please advise what is the best way to add the type as I tried

else if "mgmt" in [dn] {
grok {
match => { "[dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}mgmt%{DATA:mgmt}]%{GREEDYDATA}" }
}
mutate {
add_field => { "interface" => "%{node}/mgmt-%{mgmt}" }
remove_field => ["node","mgmt"]

}
mutate { convert => ["interface", "text"] }

}

But it did not solve the problem.

Thanks again.

MC

magnusbaeck · March 9, 2018, 8:10pm

Refresh the field list in Kibana.

micruzz82 · March 9, 2018, 9:46pm

Thanks for that @magnusbaeck I assumed that was automatically done from the Logstash side rather than from Kibana. The other mistake I made was I specified "text" while the correct one should be

mutate { convert => ["interface", "string"] }

Thank you again. till my next road block!

micruzz82 · March 9, 2018, 9:57pm

Correction: convert => {"interface" => "string"}

system · April 6, 2018, 9:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.