Unnesting user_agent fields

Elastic Stack Beats
1

After using user_agent processor, I am getting the results somewhat like this:
"user_agent" : {
"patch" : "1",
"major" : "5",
"minor" : "1",
"os" : "Android 5.1.1",
"os_minor" : "1",
"os_major" : "5",
"name" : "Android",
"os_name" : "Android",
"device" : "AFTT"
}

I want the fields inside user_agent to be split and parsed so that the JSON document looks like this
"user_agent" : {
"patch" : "1",
"major" : "5",
"minor" : "1",
"os" : "Android 5.1.1",
"os_minor" : "1",
"os_major" : "5",
"name" : "Android",
"os_name" : "Android",
"device" : "AFTT"
},
"patch" : "1",
"major" : "5",
"minor" : "1",
"os" : "Android 5.1.1",
"os_minor" : "1",
"os_major" : "5",
"name" : "Android",
"os_name" : "Android",
"device" : "AFTT"

2

Do you mind pointing out what is the difference between the different versions you have pasted here?

3

I want to bring all the fields inside user_agent field outside.

4

That is, rather than the field being user_agent.os, I want it flattened. os. not nested.

5

Well, the user agent is dissected by the User Agent processor, which has a target_field option - the default being "user-agent".
I'm not sure whether you can configure it to use the root event? Maybe by specifying an empty target_field?

6

Did that, which lead to:

{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "exception",
            "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: path cannot be null nor empty",
            "header" : {
              "processor_type" : "user_agent"
            }
          }
        ],
        "type" : "exception",
        "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: path cannot be null nor empty",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "java.lang.IllegalArgumentException: path cannot be null nor empty",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "path cannot be null nor empty"
          }
        },
        "header" : {
          "processor_type" : "user_agent"
        }
      }
    }
  ]
}
7

Sorry, I misread the title of your post.
Are you using any Filebeat modules? If not, this question should be moved to Elasticsearch subforum.

8

Yes, I am using Filebeat.

9

I meant that if have enabled any module of Filebeat.

10

I am using ingest node to read log lines. Specifically, user agent processor.

11

I tried using rename processor and it creates large number of blocks. I am looking for some shorter way to move all those fields out of user_agent field.

12

In this case, you will have better luck posting your question in the Elasticsearch forum: https://discuss.elastic.co/c/elasticsearch They develop Ingest node, so they will be able to help you. :slight_smile:

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.