# Useragent filter plugin not working for subfields?

**URL:** <https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571>\
**Category:** Logstash\
**Created:** [August 22, 2018, 2:22pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571 "2018-08-22T14:22:15Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![prprt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prprt/32/34704_2.png) [@prprt](https://discuss.elastic.co/u/prprt)\
**Post date:** [August 22, 2018, 2:22pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/1 "2018-08-22T14:22:15Z")

</div>

I have the following Useragent filter plugin configuration in Logstash 6.3.2 which does work for my Apache logs, but not for Nginx or HAProxy logs. I have double checked that the referenced field names are correct.

My Apache events are tagged as "apache\_access" and have the user agent string set in "agent". My Nginx events are tagged as "nginx\_access" and have the user agent string set in "http.agent", and my HAProxy events tagged as "haproxy" are using the exact same field "http.agent".

Why is the config below only working for Apache logs, and not working for the "[http][agent]" source? Am I making a mistake in the filter plugin configuration? Is this a bug?

```auto
filter {
  if "apache_access" in [tags] {
    useragent {
      source => "agent"
      target => "useragent"
    }
  }
  if "nginx_access" in [tags] or "haproxy" in [tags] {
    useragent {
      source => "[http][agent]"
      target => "useragent"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 22, 2018, 2:51pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/2 "2018-08-22T14:51:27Z")

</div>

Is it really a nested field or do you have a . in the field name?

---

<div class="post-metadata">

**Author:** ![prprt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prprt/32/34704_2.png) [@prprt](https://discuss.elastic.co/u/prprt)\
**Post date:** [August 23, 2018, 9:26am UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/3 "2018-08-23T09:26:25Z")

</div>

In case of Nginx logs the field comes from the following filter:

```auto
   mutate {
      rename => {
        "[json][agent]" => "http.agent"
      }

```

In case of HAproxy logs we use a Grok pattern, like this:

```auto
%{DATA:http.agent}

```

This looks indeed like field names with a dot in the name. Is that even possible? So, these are not nested fields, but simply fields with dots in their name?

I hope I have just made a simple mistake. 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 23, 2018, 12:40pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/4 "2018-08-23T12:40:38Z")

</div>

> [@prprt](#):
>
> This looks indeed like field names with a dot in the name. Is that even possible?

It is not supported, but it works right up to the corner case where it stops working.

---

<div class="post-metadata">

**Author:** ![prprt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prprt/32/34704_2.png) [@prprt](https://discuss.elastic.co/u/prprt)\
**Post date:** [August 23, 2018, 1:11pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/5 "2018-08-23T13:11:52Z")

</div>

Alright. So I need to refactor some of the Logstash filters to use proper nested fields, i.e. `rename => { "[json][agent]" => "[http][agent]" }`. Apparently Grok doesn't support nested fields, so we'll need to find alternatives for that as well.

Will this conflict with existing Elasticsearch indexes? Or are fields with dots in their name already converted to nested fields in Elasticsearch?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 23, 2018, 1:31pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/6 "2018-08-23T13:31:51Z")

</div>

> [@prprt](#):
>
> Apparently Grok doesn't support nested fields, so we'll need to find alternatives for that as well.

Instead of doing a rename of every field, you could look at the de\_dot filter using the nested option. Do note the caveats in the documentation however.

grok filters do support the nested field notation using [outer][inner]

Fields with dots in the name are indexed as such, they are not converted by elasticsearch.

---

<div class="post-metadata">

**Author:** ![prprt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prprt/32/34704_2.png) [@prprt](https://discuss.elastic.co/u/prprt)\
**Post date:** [August 23, 2018, 1:56pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/7 "2018-08-23T13:56:27Z")

</div>

> [@Badger](#):
>
> Fields with dots in the name are indexed as such, they are not converted by elasticsearch.

Does it make any difference in performance when using nested fields in Elasticsearch compared to flat fields? We were simply using "nested fields" to keep fields a bit more organized in ES, i.e. all HTTP-related data in "http.subname" fields. Does it make sense to you to use nested fields for this use case?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 23, 2018, 2:34pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/8 "2018-08-23T14:34:42Z")

</div>

It makes sense to me to use nested fields like that. I cannot speak to the performance question. You might want to ask in the elasticsearch category if there is any performance impact from nesting fields.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 20, 2018, 2:34pm UTC](https://discuss.elastic.co/t/useragent-filter-plugin-not-working-for-subfields/145571/9 "2018-09-20T14:34:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.