Useragent filter not working as expected after enabling ECS

[flalar]

We're having trouble with the useragent filter not adding the data to the document sent to Elasticsearch or stdout. Seems this happend after enabling support for ECS. Upgrading from Logstash 7.17.9 to 8.6.2 did not solve the issue either.

Filter is configured likt this

 useragent {
            source => "user_agent_original"           
            add_tag => ["useragent_added"]            
        }

example of the source "user_agent_original":"Mozilla/5.0 (Linux; Android 13; SM-G781B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36",

The tag "useragent_added" is found in the output, but no traces of the useragent

Any tips on how to debug this?

[leandrojmp (Leandro Pereira)]

@flalar

We're having trouble with the useragent filter not adding the data to the document sent to Elasticsearch or stdout.

What trouble? You need to share logs or some kind of evidence, it is not possible to know what is happening with just that information. What do you have in logstash logs? Any WARN or ERROR logs?

@flalar

The tag "useragent_added" is found in the output, but no traces of the useragent

Can you share your output? Also, share your full logstash configuration if possible.

[Rios]

Your code is OK.

   useragent {
            source => "user_agent_original"
            add_tag => ["useragent_added"]
        }

LS 8.6.2 will produce :

"user_agent" => {
       "os" => {
       "name" => "Android",
       "version" => "13",
       "full" => "Android 13"
        },
       "name" => "Chrome Mobile",
       "device" => {
            "name" => "Samsung SM-G781B"
        },
        "version" => "107.0.0.0"
    }

You can use this notation, and will get full ECS user_agent structure

   useragent {
            source => "[user_agent][original]" # instead of user_agent_original
            target => "user_agent" # the user_agent field name is default and madatory for ECS
            ecs_compatibility => "v8" # v8 is default 
            add_tag => ["useragent_added"]
        }

Will get as full ECS:

    "user_agent" => {
        "original" => "Mozilla/5.0 (Linux; Android 13; SM-G781B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36",
              "os" => {
               "name" => "Android",
            "version" => "13",
               "full" => "Android 13"
        },
            "name" => "Chrome Mobile",
          "device" => {
            "name" => "Samsung SM-G781B"
        },
         "version" => "107.0.0.0"
    }

With ecs_compatibility => "disabled" , result will be:

    "user_agent" => {
             "patch" => "0",
           "os_full" => "Android 13",
          "original" => "Mozilla/5.0 (Linux; Android 13; SM-G781B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Mobile Safari/537.36",
             "major" => "107",
                "os" => "Android",
             "minor" => "0",
        "os_version" => "13",
          "os_major" => "13",
              "name" => "Chrome Mobile",
           "os_name" => "Android",
           "version" => "107.0.0.0",
            "device" => "Samsung SM-G781B"
    }