JSON parsing creates a field for each user

Apologies if it's a silly question but I googled around and I couldn't find a solution, I don't have much experience with Elasticsearch.
I have an ES 5.5 stack deployed over the AWS ES service, and I have some JSON logs that I would like to parse.
My problem is that ES adds a users.user1 field, so now I have hundreds of those and I cannot select them manually when creating a dashboard because they're way too many.

"_index": "cwl-2018.08.05",
"_type": "/aws/lambda/tagtest",
"_id": "34197741497112809064849409673362766816549106281078849536",
"_version": 1,
"_score": 1,
"_source": {
"users": {
"user1": [
"@id": "34197741497112809064849409673362766816549106281078849536",
"@timestamp": "2018-08-05T14:36:04.585Z",
"@message": "{"users": {"user1": ["tag1", "tag2", "tag3", ]}}\n",
"@owner": "024533801171",
"@log_group": "/aws/lambda/tagtest",
"@log_stream": "2018/08/05/[$LATEST]5f7a5a39fcbc4e08b0c011bbf7d5112d"
"fields": {
"@timestamp": [

What I would need is a CSV with three columns: the usernames, the number of times these usernames were found in the logs, and the tags associated with each username (also, an username can have a tag1 in one log and a tag2 in another log, I would need both tags)

Anyone knows how can I achieve this in Kibana? Thank you,

Does each log have only one Username inside the users array?

You will not be able to get the o/p you desire (CSV) directly, till you probably normalize your logs using some ETL like Logstash.

Thank you for the reply, no, currently the number of users in each log can vary, but if that's a problem since I control the application I might be able to modify the JSON to send a single user per log. Any idea if I could accomplish it using any kind of aggregation? The log looks like it's being parsed and the JSON is RFC compliant, so I thought that there was a way using the Kibana data table dashboards and then export as CSV.

No. Actually the value items are coming as keys. Therefore, aggregation will not be possible. If the fields or logs were in the following format, then it would be properly indexed:

  "users" : "john",
  "tags": ["tag1", "tag2"]
1 Like

Thank you for your help. I changed the log format of the application to match your format and I can see that the users are now being parsed as values. I just have one doubt on how to proceed, I'm configuring my aggregations as follow:


This gives me several lines of the tag and the corresponding user, but apparently it doesn't report the users without tags, such as:
"users" : "john",
"tags" :
Any idea how should I configure the aggregation in this case to display also the users not associated with any tag? Thank you again for your help, I really appreciate it

Create a simple data table based on either unique count or count depending on requirement. Display the columns needed and then use a does not exist filter for the tag field.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.