# What is the difference between fields pass by filebeat and fields in logstash?

**URL:** <https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727>\
**Category:** Logstash\
**Created:** [May 22, 2021, 2:50pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727 "2021-05-22T14:50:14Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 22, 2021, 2:50pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/1 "2021-05-22T14:50:14Z")

</div>

I'm a new guy about elk. i'm trying use logstash filter instead of filebeat processors.

There are some fields passed by filebeat in logstash , for example: i had set some fields in filebeat:  
"fields.app"="nginx" or "docker"  
"fields.localtion"="hk" or "us"  
"docker.attrs.tag"="xxxx" which passed by docker with `"log-opts":{"tag": "{{.Name}}"}` in /etc/docker/daemon.json

Now i can use `if "xxx" in [xxxx][xxx]` or `if [xxxx][xxxx][xxx] =~ "filebeat"`just like following:

```
if [docker][attrs][tag] =~ "filebeat" {
        drop {}
    }

```

It's working good.

But,when i use filter plugin to add new fields in logstash. i can't use if [xxx][xxx] to fliter something.  
for example:  
I use grok match the nginx access log to get `"%{QS:useragent}"` and use `useragent` to get some fields.

```
useragent {
            source => "useragent"
            target => "client.agent"
        }

```

I got some filed like `client.agent.name` `client.agent.device` and more.

 ![截屏2021-05-22 22.55.37](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c7217e9ba19e08cb4483138b06717613da8eaf2.png)

Now when i set filter `if [client][agent][device] =~ "Spider" { drop {} }` it doesn't work.  
I don't kown why....

Another question about fields syntax:  
What is the difference between `[docker][attrs][tag]` and `docker.attrs.tag` ? sometime `docker.attrs.tag` is worked sometime does not worked.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 22, 2021, 3:14pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/2 "2021-05-22T15:14:31Z")

</div>

> [@stille](#):
>
> `target => "client.agent"`

That creates fields inside the object [client.agent]. So you have fields like [client.agent][device]. Most likely you want to use

```
target => "[client][agent]"

```

---

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 22, 2021, 3:42pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/3 "2021-05-22T15:42:29Z")

</div>

Thanks for help  
It's working now.

So about another question mayby i knew it.  
`clients.agent`In logstash also is `[clients.agent]`will be show `clients.agent`on kibana  
`[client][agent]` in logstash will be show `clients.agent`on kibana too.  
Turns out i confuse the two...  
It's correct?

---

<div class="post-metadata">

**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\
**Post date:** [May 22, 2021, 11:57pm UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/4 "2021-05-22T23:57:33Z")

</div>

You may be less confused if you view the data in Kibana (in the Discover app) as JSON; this will strip away some of the presentation layer that Kibana will be doing.

`[client][agent] = "foo"` in Logstash will be the same as the following in JSON

```auto
{"client":
    {"agent": "foo"}
}

```

Kibana will display this as `client.agent`, which is why field names mustn't (shouldn't?) have dots in them.

I'm not certain what Logstash would make of the `[client.agent][device] = "Spider"`, but I imagine it would probably result in the following JSON and **should be avoided**

```auto
{"client.agent":
    {"device": "Spider"}
}

```

On a similar point of confusion, note that Kibana will use '-' to indicate a null field. This is not the same as when the likes of Apache httpd puts a '-' in its access log to indicate a null field; in that case, without processing to strip it out, it will be exactly the string '-'. This is something that viewing records as JSON can make clearer.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 23, 2021, 1:57am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/5 "2021-05-23T01:57:10Z")

</div>

> [@cknz](#):
>
> I'm not certain what Logstash would make of the `[client.agent][device] = "Spider"` , but I imagine it would probably result in the following JSON and **should be avoided**
> 
> ```auto
> {"client.agent":
> {"device": "Spider"}
> }
> 
> ```

I am certain 😃 and that is exactly what it will do. logstash has no problem with periods in object or field names because its syntax is not ambiguous.

elasticsearch originally allowed it, then banned it, then patched the parser to provide error messages that made the conflict between the two clearer when you tried to use both, so it reënabled it.

Periods in field and object names are a terrible idea, but the product allows you to implement bad ideas.

---

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 23, 2021, 3:37am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/6 "2021-05-23T03:37:35Z")

</div>

@cknz @Badger

Thanks both of you again .

I used the period as a subfield delimite.

I checked oldest logs as JSON , you're right.`client.agent`is a field, is not a subfield.

![截屏2021-05-23 11.20.54](https://us1.discourse-cdn.com/elastic/original/3X/9/7/97d33561142aa05696ff987e9ca3c79306bba5c4.png)

Just like`[client][agent]` is the best solution.

 ![截屏2021-05-23 11.21.28](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4ce0c352706861f4ab8466f223d334bac59fb642.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2021, 3:37am UTC](https://discuss.elastic.co/t/what-is-the-difference-between-fields-pass-by-filebeat-and-fields-in-logstash/273727/7 "2021-06-20T03:37:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.