JSON parsing creates a field for each user

Elastic Stack Kibana
1

Apologies if it's a silly question but I googled around and I couldn't find a solution, I don't have much experience with Elasticsearch.
I have an ES 5.5 stack deployed over the AWS ES service, and I have some JSON logs that I would like to parse.
My problem is that ES adds a users.user1 field, so now I have hundreds of those and I cannot select them manually when creating a dashboard because they're way too many.

{
"_index": "cwl-2018.08.05",
"_type": "/aws/lambda/tagtest",
"_id": "34197741497112809064849409673362766816549106281078849536",
"_version": 1,
"_score": 1,
"_source": {
"users": {
"user1": [
"tag1",
"tag2",
"tag3"
]
},
"@id": "34197741497112809064849409673362766816549106281078849536",
"@timestamp": "2018-08-05T14:36:04.585Z",
"@message": "{"users": {"user1": ["tag1", "tag2", "tag3", ]}}\n",
"@owner": "024533801171",
"@log_group": "/aws/lambda/tagtest",
"@log_stream": "2018/08/05/[$LATEST]5f7a5a39fcbc4e08b0c011bbf7d5112d"
},
"fields": {
"@timestamp": [
1533479764585
]
}
}

What I would need is a CSV with three columns: the usernames, the number of times these usernames were found in the logs, and the tags associated with each username (also, an username can have a tag1 in one log and a tag2 in another log, I would need both tags)

Anyone knows how can I achieve this in Kibana? Thank you,

2

Does each log have only one Username inside the users array?

You will not be able to get the o/p you desire (CSV) directly, till you probably normalize your logs using some ETL like Logstash.

3

Thank you for the reply, no, currently the number of users in each log can vary, but if that's a problem since I control the application I might be able to modify the JSON to send a single user per log. Any idea if I could accomplish it using any kind of aggregation? The log looks like it's being parsed and the JSON is RFC compliant, so I thought that there was a way using the Kibana data table dashboards and then export as CSV.

4

No. Actually the value items are coming as keys. Therefore, aggregation will not be possible. If the fields or logs were in the following format, then it would be properly indexed:

{
  "users" : "john",
  "tags": ["tag1", "tag2"]
}
1 Like
5

Thank you for your help. I changed the log format of the application to match your format and I can see that the users are now being parsed as values. I just have one doubt on how to proceed, I'm configuring my aggregations as follow:

35
43

This gives me several lines of the tag and the corresponding user, but apparently it doesn't report the users without tags, such as:
{
"users" : "john",
"tags" :
}
Any idea how should I configure the aggregation in this case to display also the users not associated with any tag? Thank you again for your help, I really appreciate it

6

Create a simple data table based on either unique count or count depending on requirement. Display the columns needed and then use a does not exist filter for the tag field.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.