One doc for each array item

Elastic Stack Logstash
1

I have documents with several arrays. one array I have gotten to work with help here. Thank you very much Badger. I have a 3 arrays I want to be able to visualize , but one is most important. It is useractions array. It will sometimes have 0 records, but other times may have 1-15 records. I would like to create one document for each useraction and include all the other information in the main document including the other arrays, errors and events. I mostly need the useraction data for performance reasons.

How do most pople handle these issues? do they leave the document as is and learn to write code to visualize the nested arrays, assuming that is possible or do they flatten out the array like I want to do?

having said that, I am looking for any example code that would create a new document foreach useraction doc with all the other underlying fields and arrays as well.

Sample of obfuscated data:

{"userActions":[{"navigationStart":1591196810169,"stringProperties":"","requestStart":174,"longProperties":"","endTime":1591196834131,"duration":23962,"apdexCategory":"TOLERATING","responseStart":182,"failedXhrRequests":0,"frontendTime":23777,"visuallyCompleteTime":6283,"failedImages":0,"loadEventStart":2541,"keyUserAction":true,"userActionPropertyCount":0,"firstPartyResources":166,"application":"Client","httpRequestsWithErrors":0,"firstPartyBusyTime":22181,"dateProperties":"","speedIndex":4303,"startTime":1591196810169,"domCompleteTime":2537,"errorCount":0,"thirdPartyResources":31,"responseEnd":185,"serverTime":8,"domain":"client.abc.com","domContentLoadedTime":1936,"targetUrl":"https://client.abc.com/client/dashboards","cdnBusyTime":null,"name":"loading of page /client/dashboards","internalApplicationId":"APPLICATION-Bogus","internalKeyUserActionId":"APPLICATION_METHOD-Bogus","networkTime":177,"type":"Load","cdnResources":0,"loadEventEnd":2544,"documentInteractiveTime":1619,"doubleProperties":"","thirdPartyBusyTime":1996}],"isp":"Bogus-inetprovider","syntheticEvents":,"userType":"REAL_USER","duration":23962,"browserFamily":"Chrome","ip":"1.1.1.1","events":[{"internalApplicationId":"APPLICATION-Bogus","type":"UserTag","name":"DevInt Bogus User","startTime":1591196834131,"application":"Client"}],"@timestamp":"2020-06-05T09:34:37.202Z","screenWidth":1920,"osVersion":"Windows 10","clientType":"Desktop Browser","browserMajorVersion":"Chrome 83","startTime":1591196810169,"browserType":"Desktop Browser","endReason":"TIMEOUT","city":"Chumlee","tenantId":"Bogus1","errors":,"connectionType":"UNKNOWN","userSessionId":"Bogususerid-0","continent":"North America","screenHeight":1080,"matchingConversionGoalsCount":0,"region":"North Carolina","screenOrientation":"LANDSCAPE","stringProperties":[{"key":"usertype","value":"R","internalApplicationId":"APPLICATION-Bogus","applicationId":"Client"},{"key":"firmid","value":"100","internalApplicationId":"APPLICATION-Bogus","applicationId":"Client"},{"key":"firmname","value":"ABC","internalApplicationId":"APPLICATION-Bogus","applicationId":"Client"}],"numberOfRageClicks":0,"longProperties":,"endTime":1591196834131,"@version":"1","osFamily":"Windows","hasSessionReplay":false,"totalErrorCount":0,"dateProperties":,"bounce":true,"userExperienceScore":"TOLERATED","totalLicenseCreditCount":0,"userId":"DevInt Bogus User","newUser":true,"displayResolution":"FHD","path":"/etc/logstash/conf.d/data.txt","userActionCount":1,"host":"somehost","applicationType":"WEB_APPLICATION","country":"United States","internalUserId":"12345678","doubleProperties":}

What I would like to see would be one document for every useraction with all fields not in useractions included

2

If you have an array and want to separate it into multiple documents then use a split filter.

3

I had written one before to try and it did not work. does the syntax look correct?

split {
field => "[useractions]"
}

mutate {
add_field => {
"name" => "%{[useractions][name]}"
"domain" => "%{[useractions][domain]}"
"targeturl" => "%{[useractions][targeturl]}"
"type" => "%{[useractions][type]}"
"starttime" => "%{[useractions][starttime]}"
"endtime" => "%{[useractions][endtime]}"
"duration" => "%{[useractions][duration]}"
"application" => "%{[useractions][application]}"
"internalapplicationid" => "%{[useractions][internalapplicationid]}"
"speedindex" => "%{[useractions][speedindex]}"
"errorcount" => "%{[useractions][appVersion]}"
"apdexcategory" => "%{[useractions][apdexcategory]}"
"networktime" => "%{[useractions][networktime]}"
"servertime" => "%{[useractions][servertime]}"
"frontendtime" => "%{[useractions][frontendtime]}"
"documentinteractivetime" => "%{[useractions][documentinteractivetime]}"
"failedimages" => "%{[useractions][failedimages]}"
"failedxhrrequests" => "%{[useractions][failedxhrrequests]}"
"httprequestswitherrors" => "%{[useractions][httprequestswitherrors]}"
"thirdpartyresources" => "%{[useractions][thirdpartyresources]}"
"thirdpartybusytime" => "%{[useractions][thirdpartybusytime]}"
"firstpartyresources" => "%{[useractions][firstpartyresources]}"
"cdnResources" => "%{[useractions][cdnResources]}"
"cdnBusyTime" => "%{[useractions][cdnBusyTime]}"
"firstpartybusytime" => "%{[useractions][firstpartybusytime]}"
"domcompletetime" => "%{[useractions][domcompletetime]}"
"domcontentloadedtime" => "%{[useractions][domcontentloadedtime]}"
"loadeventstart" => "%{[useractions][loadeventstart]}"
"loadeventend" => "%{[useractions][loadeventend]}"
"navigationstart" => "%{[useractions][navigationstart]}"
"requeststart" => "%{[useractions][requeststart]}"
"responsestart" => "%{[useractions][responsestart]}"
"responseend" => "%{[useractions][responseend]}"
"visuallycompletetime" => "%{[useractions][visuallycompletetime]}"
"keyuseraction" => "%{[useractions][keyuseraction]}"
}
remove_field => [ "[useractions]" ]
}

4

this is the error I am getting.
Only String and Array types are splittable. field:[useractions] is of type = NilClass

5

That is telling you that the [useractions] field does not exist. You did parse the JSON with either a codec or a json filter, right?

6

Yes

filter {
json {
source => "message"
}
ruby {
code => '
props = event.get("stringProperties")
if props
props.each { |x|
key = x["key"]
event.set("stringProperties.#{key}", x["value"])
}
event.remove("stringProperties")
end
'
}
split {
field => "useractions"
}

mutate {
add_field => {
"name" => "%{[useractions][name]}"
"domain" => "%{[useractions][domain]}"
"targeturl" => "%{[useractions][targeturl]}"
"type" => "%{[useractions][type]}"
"starttime" => "%{[useractions][starttime]}"
"endtime" => "%{[useractions][endtime]}"
"duration" => "%{[useractions][duration]}"
"application" => "%{[useractions][application]}"
"internalapplicationid" => "%{[useractions][internalapplicationid]}"
"speedindex" => "%{[useractions][speedindex]}"
"errorcount" => "%{[useractions][appVersion]}"
"apdexcategory" => "%{[useractions][apdexcategory]}"
"networktime" => "%{[useractions][networktime]}"
"servertime" => "%{[useractions][servertime]}"
"frontendtime" => "%{[useractions][frontendtime]}"
"documentinteractivetime" => "%{[useractions][documentinteractivetime]}"
"failedimages" => "%{[useractions][failedimages]}"
"failedxhrrequests" => "%{[useractions][failedxhrrequests]}"
"httprequestswitherrors" => "%{[useractions][httprequestswitherrors]}"
"thirdpartyresources" => "%{[useractions][thirdpartyresources]}"
"thirdpartybusytime" => "%{[useractions][thirdpartybusytime]}"
"firstpartyresources" => "%{[useractions][firstpartyresources]}"
"cdnResources" => "%{[useractions][cdnResources]}"
"cdnBusyTime" => "%{[useractions][cdnBusyTime]}"
"firstpartybusytime" => "%{[useractions][firstpartybusytime]}"
"domcompletetime" => "%{[useractions][domcompletetime]}"
"domcontentloadedtime" => "%{[useractions][domcontentloadedtime]}"
"loadeventstart" => "%{[useractions][loadeventstart]}"
"loadeventend" => "%{[useractions][loadeventend]}"
"navigationstart" => "%{[useractions][navigationstart]}"
"requeststart" => "%{[useractions][requeststart]}"
"responsestart" => "%{[useractions][responsestart]}"
"responseend" => "%{[useractions][responseend]}"
"visuallycompletetime" => "%{[useractions][visuallycompletetime]}"
"keyuseraction" => "%{[useractions][keyuseraction]}"
}
remove_field => [ "[useractions]" ]
}
}

7

I am almost home. I found the issue with field name , I needed an uppercase A i had a lower case a.

My only remaining issue is I get this output "targeturl" : "%{[userActions][targeturl]}",
Not using the right % something

split {
field => "[userActions]"
}
mutate {
add_field => {
"name" => "%{[userActions][name]}"
"domain" => "%{[userActions][domain]}"
"targeturl" => "%{[userActions][targeturl]}"

8

It should be targetUrl, not targeturl.

If you want to move every field inside userActions to the top level it might be easier to do it like this, in a ruby filter.

9

somewhere along the line I went to lower case. Trying that now. Thanks. I should have caught that.

10

I finally got it to work. I still have one issue but I am posting a different thread for that. Thank you again Badger.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.