How to split and label fields of an array of arrany?


(Chuan Ydnk) #1

Hi

I have the following output after i had used filter { json { source => "message" } }

Now, I wanted to split the array of objects

The first field is link_name, link_alias, bto,bcs,bss.

I am sorry really a newbie in logstash

"formattedData":[["10.212.75.73 WAN optimized","10.212.75.73 WAN optimized","5.0254744E7","2.70668E7","2.3187946E7"],["10.212.75.73 WAN pass through","10.212.75.73 WAN pass through","4.5109216E7","1.1213042E7","3.3896176E7"],["172.18.66.5 wan0_0","BJS11ACC001-WAN0_0","9149226.0","1973337.125","7175889.0"],["57.28.59.5 wan0_0","BOM11ACC001-WAN0_0","6082061.0","3228388.0","2853673.0"],["57.28.84.5 wan0_0","BOM41ACC001-WAN0_0","5317941.5","3211377.25","2106564.25"],["172.18.80.5 wan0_0","CAN41ACC001-WAN0_0","1.6712449E7","1.0740312E7","5972137.0"],["172.28.11.82 wan2_0","HKGX3ACC0001-WAN2_0","9.0989192E7","4.2567948E7","4.842124E7"],["172.28.11.82 wan3_0","HKGX3ACC0001-WAN3_0","3.3666396E7","1.3425408E7","2.024099E7"],["10.164.6.28 GigabitEthernet0/0/1","PMUC2345-WAN0_0-GE0_0_1","1.9228112E7","2586036.75","1.6642074E7"],["57.28.18.22 wan0_0","TPE11ACC001-WAN0_0","8087264.5","1268966.75","6818298.0"]]


(Chuan Ydnk) #2

Hi there,

i need some help here, i manage to use spit and mutate to achieve "nearly" what i wanted but however, the [formattedData][1] dosen't return me the correct value. What is it so ?

input {
http_poller {
urls => {
test1 => {
url => "https://wecacascxv01.assurance.sita.aero/rest/dmiquery/getDMIData3"
method => post
user => "xxxxx"
password => "xxxxx"
headers => {
"Content-Type" => "application/json"
}
body => '{"appId": "CVENT","viewId": "TfcOnLink","dataSourceId": "ALL_AGGR","dimensionIds": ["msrPoint","msrPoint4Cust"],"metricIds": ["BTO","BCS","BSC"],"dimFilters": [["msrPoint4Cust","WAN",false]],"metricFilters": [["BTO",">",5000000,1]],"sort": ,"top": 1000,"resolution": "r","timePeriod": "p","numberOfPeriods": 1}'
}
}
request_timeout => 60
schedule => { "every" => "60s" }
# A hash of request metadata info (timing, response headers, etc.) will be sent here
# metadata_target => "http_poller_metadata"
tags => dcrum_endpoints
}
}

filter {
json {
source => "message"
}
split {
field => "[formattedData]"
}
split {
field => "[formattedData][0]"
}
mutate {
remove_field => ["rawData","dimensionIds","metricIds","columnHeader","columnUnit","columnRendererIds","response_message","columnHeaderName"]
}
mutate {
rename => { "[formattedData][0]" => "Link name" }
rename => { "[formattedData][1]" => "Link alias" }
rename => { "[formattedData][2]" => "Total bandwidth usage" }
rename => { "[formattedData][3]" => "Incoming bandwidth usage" }
rename => { "[formattedData][4]" => "Outgoing bandwidth usage" }
rename => { "[formattedData][5]" => "Outgoing bandwidth usage5" }
rename => { "[formattedData][6]" => "Outgoing bandwidth usage6" }
rename => { "[formattedData][7]" => "Outgoing bandwidth usage7" }
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "wecascxsv01-json-%{+YYYY.MM.dd}"
}
file {
path => "/tmp/debug-%{+YYYY.MM.dd}.txt"
}
stdout {
codec => rubydebug
}

Here's the output:

I don't understand why [formattedData][1] should be return the BJS11ACC001-WAN0_) but it returned the Total bandwidth attribute which is the third attribute of the formattedData ? anyone can explain ?

{"@version":"1","formattedData":["BJS11ACC001-WAN0_0","2770949.75"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"172.18.66.5 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"7976346.5","Total bandwidth usage":"5205396.5","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["BOM41ACC001-WAN0_0","3475577.75"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"57.28.84.5 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"5288093.0","Total bandwidth usage":"1812515.125","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["CAN41ACC001-WAN0_0","5619251.5"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"172.18.80.5 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"9131968.0","Total bandwidth usage":"3512716.75","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["HKGX3ACC0001-WAN2_0","4.6233968E7"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"172.28.11.82 wan2_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"9.148968E7","Total bandwidth usage":"4.5255716E7","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["HKGX3ACC0001-WAN3_0","1.0402631E7"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"172.28.11.82 wan3_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"3.1911948E7","Total bandwidth usage":"2.1509318E7","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["LON11ACC001-WAN0_0","1257672.5"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"57.28.25.5 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"1.192677E7","Total bandwidth usage":"1.0669098E7","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["PMUC2345-WAN0_0-GE0_0_1","2494789.75"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"10.164.6.28 GigabitEthernet0/0/1","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"1.6676879E7","Total bandwidth usage":"1.4182089E7","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["SHA11ACC001-WAN0_0","7592502.5"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"172.18.65.5 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"1.0733222E7","Total bandwidth usage":"3140720.0","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["TPE11ACC001-WAN0_0","1477641.875"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"57.28.18.22 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"8822487.0","Total bandwidth usage":"7344845.5","@timestamp":"2018-11-12T08:13:01.755Z"}
{"@version":"1","formattedData":["TYO11ACC001-WAN0_0","2499608.5"],"timeBegin":1542009900000,"timeout":false,"tags":["dcrum_endpoints","_grokparsefailure","_geoip_lookup_failure"],"timeEnd":1542010200000,"Link name":"57.28.48.5 wan0_0","dmiServiceError":[{"error":false,"info":false,"warning":false}],"timeoutValue":600000,"Link alias":"5050072.0","Total bandwidth usage":"2550463.75","@timestamp":"2018-11-12T08:13:01.755Z"}


(Chuan Ydnk) #3

Any kind souls able to help ?


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.