# Help with logstash json output to split a field in multiple variables and combine them to a new field

**URL:** <https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592>\
**Category:** Logstash\
**Created:** [March 5, 2018, 8:33pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592 "2018-03-05T20:33:12Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 5, 2018, 8:33pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/1 "2018-03-05T20:33:12Z")

</div>

Hi All

I have this JSON output as below.

{ "totalCount": "1", "imdata": [{ "l1PhysIf": { "attributes": { "adminSt": "up", "autoNeg": "on", "brkoutMap": "none", "bw": "0", "childAction": "", "delay": "1", "descr": "srv", "dn": "topology/pod-1/node-403/sys/phys-[eth1/1]", "dot1qEtherType": "0x8100", "ethpmCfgFailedBmp": "", "ethpmCfgFailedTs": "00:00:00:00.000", "ethpmCfgState": "0", "fecMode": "inherit", "id": "eth1/1", "inhBw": "unspecified", "layer": "Layer2", "lcOwn": "local", "linkDebounce": "100", "linkLog": "default", "mdix": "auto", } } } ] }

The field I am interested in is "dn": "topology/pod-1/node-403/sys/phys-[eth1/1]".

and I would like to extract this and combine the values into a new field as "interface" =\> 403/1/1

And when running a query in elasticsearch for "interface" and "linkDebounce", I should get the output as 100.

Would anyone be able to help me on how to go about this please if you have experience in this.

Thanks very much.

MC

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 5, 2018, 8:49pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/2 "2018-03-05T20:49:46Z")

</div>

to create a new field you can use mutate add\_field. but, from where you are getting "403/1/1". hoping 403 from dn and what about 1/1 ? end part of dn or static ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 5, 2018, 8:54pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/3 "2018-03-05T20:54:35Z")

</div>

Your example isn't valid JSON but perhaps you have simplified it.

Does the `imdata` always contain an array with a single element like in this case?

If yes, use a grok filter to parse the `[imdata][0][l1PhysIf][attributes][dn]` field and extract the interesting pieces. You can then use `add_field` in the same grok filter or in a separate mutate filter to piece together the different field values that you're picking up from different parts of the string.

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 5, 2018, 9:12pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/4 "2018-03-05T21:12:51Z")

</div>

> [@micruzz82](#):
>
> { "totalCount": "1", "imdata": [{ "l1PhysIf": { "attributes": { "adminSt": "up", "autoNeg": "on", "brkoutMap": "none", "bw": "0", "childAction": "", "delay": "1", "descr": "srv", "dn": "topology/pod-1/node-403/sys/phys-[eth1/1]", "dot1qEtherType": "0x8100", "ethpmCfgFailedBmp": "", "ethpmCfgFailedTs": "00:00:00:00.000", "ethpmCfgState": "0", "fecMode": "inherit", "id": "eth1/1", "inhBw": "unspecified", "layer": "Layer2", "lcOwn": "local", "linkDebounce": "100", "linkLog": "default", "mdix": "auto" } } } ] }

Hi Magnus, yes I did truncate it, I have removed the last comma after auto so that it is a valid JSON.

However that was only one instance of an API query.

Here is another query output for an interface.

{  
"totalCount": "1",  
"imdata": [  
{  
"eqptIngrTotal5min": {  
"attributes": {  
"bytesAvg": "11464546",  
"bytesBase": "0",  
"bytesCum": "19503139467272",  
"bytesLast": "12658946",  
"bytesMax": "12933282",  
"bytesMin": "9452679",  
"bytesPer": "45858186",  
"bytesRate": "1146454.650000",  
"bytesRateAvg": "1144584.823232",  
"bytesRateLast": "1150813.272727",  
"bytesRateMax": "1201475.444444",  
"bytesRateMin": "1050297.666667",  
"bytesRateSpct": "0",  
"bytesRateThr": "",  
"bytesRateTr": "0.000000",  
"bytesRateTrBase": "1147684.594554",  
"bytesRateTtl": "4578339.292929",  
"bytesSpct": "0",  
"bytesThr": "",  
"bytesTr": "0",  
"bytesTrBase": "346504702",  
"childAction": "",  
"cnt": "4",  
"dn": "topology/pod-1/node-103/sys/phys-[eth1/1]/CDeqptIngrTotal5min",  
"lastCollOffset": "40",  
"modTs": "never",  
"pktsAvg": "10995",  
"pktsBase": "0",  
"pktsCum": "12823322256",  
"pktsLast": "8292",  
"pktsMax": "18740",  
"pktsMin": "6921",  
"pktsPer": "43983",  
"pktsRate": "1099.575000",  
"pktsRateAvg": "1085.224747",  
"pktsRateLast": "753.818182",  
"pktsRateMax": "1703.636364",  
"pktsRateMin": "753.818182",  
"pktsRateSpct": "0",  
"pktsRateThr": "",  
"pktsRateTr": "0.000000",  
"pktsRateTrBase": "795.597602",  
"pktsRateTtl": "4340.898990",  
"pktsSpct": "0",  
"pktsThr": "",  
"pktsTr": "0",  
"pktsTrBase": "236893",  
"repIntvEnd": "2018-03-05T21:10:29.990+00:00",  
"repIntvStart": "2018-03-05T21:09:49.990+00:00",  
"status": "",  
"utilAvg": "0",  
"utilLast": "0",  
"utilMax": "0",  
"utilMin": "0",  
"utilSpct": "0",  
"utilThr": "",  
"utilTr": "0",  
"utilTrBase": "0",  
"utilTtl": "0"  
}  
}  
}  
]  
}

@tag_v the 1/1 was part of the dn field which had eth1/1

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 5, 2018, 9:32pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/5 "2018-03-05T21:32:04Z")

</div>

in this scenario as magnus mentioned you need to choose grok patterns to extract node number and eth port of "dn" field. (Assuming imdata array has only 1 dn field)

```
filter {
if [imdata][0][l1PhysIf][attributes][dn] {
	grok {
		match => { "[imdata][0][l1PhysIf][attributes][dn]" => "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}\]%{GREEDYDATA}" }
	}
	mutate {
		add_field => { "interface" => "%{node}/%{eth}" }
		remove_field => ["node","eth"]
	}
}
}
```

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 5, 2018, 9:39pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/6 "2018-03-05T21:39:55Z")

</div>

> [@tag\_v](#):
>
> match =\> { "[imdata][0][l1PhysIf][attributes][dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}]%{GREEDYDATA}" }

Thank you @tag_v and @magnusbaeck

I have run a few more API queries and yes it imdata only has 1 dn field in it.

I will test your filter and see how we can port them into elasticsearch. This will give us very good information on interface stats and hopefully we can build good visualizations for our monitoring.

Many thanks again team.

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 9, 2018, 2:32pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/7 "2018-03-09T14:32:43Z")

</div>

I had another question from the above, I was trying to build a conditional filter for the 3 values as below.

topology/pod-1/node-104/sys/phys-[eth1/6]/CDeqptIngrBytes5min

topology/pod-1/node-101/sys/aggr-[po6]/CDeqptIngrBytes5min

topology/pod-1/node-101/sys/mgmt-[mgmt0]/CDeqptIngrBytes5min

This is my filter.

filter {  
if [imdata][0][eqptIngrBytes5min][attributes][dn] {

if [aggr] in "[imdata][0][eqptIngrBytes5min][attributes][dn]" {  
grok {  
match =\> { "[imdata][0][eqptIngrBytes5min][attributes][dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}po%{DATA:po}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/po-%{po}" }  
remove\_field =\> ["node","po"]  
}

}

else if [mgmt] in "[imdata][0][eqptIngrBytes5min][attributes][dn]" {  
grok {  
match =\> { "[imdata][0][eqptIngrBytes5min][attributes][dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}mgmt%{DATA:mgmt}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/mgmt-%{mgmt}" }  
remove\_field =\> ["node","mgmt"]  
}

}

else {  
grok {  
match =\> { "[imdata][0][eqptIngrBytes5min][attributes][dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/%{eth}" }  
remove\_field =\> ["node","eth"]  
}

}

}  
}

However it doesn't seem to be working.

Can you advise what I need to change from the filter so that the 3 variables can match accordingly and set the interface to the appropriate value.

Thanks for your help

Regards

MC

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 9, 2018, 3:14pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/8 "2018-03-09T15:14:34Z")

</div>

I figured it out in the end for those it may help.

filter {  
if [dn] {

if "aggr" in [dn] {  
grok {  
match =\> { "[dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}po%{DATA:po}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/portchannel-%{po}" }  
remove\_field =\> ["node","po"]  
}

}

else if "mgmt" in [dn] {  
grok {  
match =\> { "[dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}mgmt%{DATA:mgmt}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/mgmt-%{mgmt}" }  
remove\_field =\> ["node","mgmt"]  
}

}

else {  
grok {  
match =\> { "[dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}eth%{DATA:eth}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/%{eth}" }  
remove\_field =\> ["node","eth"]  
}

}

}  
}

Thanks for @tag_v and @magnusbaeck for the initial guidance on this.

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 9, 2018, 4:40pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/9 "2018-03-09T16:40:58Z")

</div>

I just had one more query to find the correct way to add the data type.

Right now I see a ? in regards to the type.

![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2817d3a12527b8a48b127d37f74cae3fcdc18a8.png)

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/0/40ee87aa52ae11aea3f67491edd7d9c65b094b77.png)

![image](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b7b73a79b4af29771cc52d105edfab7ee14ee87c.png)

Could you please advise what is the best way to add the type as I tried

else if "mgmt" in [dn] {  
grok {  
match =\> { "[dn]" =\> "%{GREEDYDATA}node-%{DATA:node}/%{GREEDYDATA}mgmt%{DATA:mgmt}]%{GREEDYDATA}" }  
}  
mutate {  
add\_field =\> { "interface" =\> "%{node}/mgmt-%{mgmt}" }  
remove\_field =\> ["node","mgmt"]

```
}
mutate { convert => ["interface", "text"] }

```

}

But it did not solve the problem.

Thanks again.

MC

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 9, 2018, 8:10pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/10 "2018-03-09T20:10:30Z")

</div>

Refresh the field list in Kibana.

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 9, 2018, 9:46pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/11 "2018-03-09T21:46:41Z")

</div>

Thanks for that @magnusbaeck I assumed that was automatically done from the Logstash side rather than from Kibana. The other mistake I made was I specified "text" while the correct one should be

mutate { convert =\> ["interface", "string"] }

Thank you again. till my next road block!

---

<div class="post-metadata">

**Author:** ![micruzz82](https://avatars.discourse-cdn.com/v4/letter/m/db5fbb/32.png) [@micruzz82](https://discuss.elastic.co/u/micruzz82)\
**Post date:** [March 9, 2018, 9:57pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/12 "2018-03-09T21:57:10Z")

</div>

Correction: convert =\> {"interface" =\> "string"}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2018, 9:57pm UTC](https://discuss.elastic.co/t/help-with-logstash-json-output-to-split-a-field-in-multiple-variables-and-combine-them-to-a-new-field/122592/13 "2018-04-06T21:57:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.