How to create a pipeline on a field that contains dots

When I use /_ingest/pipeline/user_agent to create a pipeline as such:

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "meta.http.headers.user-agent",
                "ignore_missing": true
            }
        }
    ]
}

What I really want is to allow the pipeline to process data below:

{
    "meta": {
        "http.headers.user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

I also tried to create pipeline as below:

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "meta.http\\.headers\\.user-agent",
                "ignore_missing": true
            }
        }
    ]
}

It wouldn't work.

When I have pipeline as such:

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "meta.user-agent",
                "ignore_missing": true
            }
        }
    ]
}

Whilst the data is

{
    "meta": {
        "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

It works.

Unfortunately our architecture has already set-up to produce data like

{
    "meta": {
        "http.headers.user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

It's pretty much impossible to update all of our apps and services to change the data structure at this point... :frowning: So is it possible to support

{
    "meta": {
        "http.headers.user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

and how?

Hi @aeroxy
Yup that's a problem you need to use the dot expander processor first.

How do I know? ... because I stayed up half a night until someone showed me :slight_smile:

Take a look and try it... let us know

" Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor."

So run the dot expander first .. then you access the field the way you want but the underlying json will have the correct object hierarchy see the examples it will help.