What is the difference between fields pass by filebeat and fields in logstash?

[stille]

I'm a new guy about elk. i'm trying use logstash filter instead of filebeat processors.

There are some fields passed by filebeat in logstash , for example: i had set some fields in filebeat:
"fields.app"="nginx" or "docker"
"fields.localtion"="hk" or "us"
"docker.attrs.tag"="xxxx" which passed by docker with "log-opts":{"tag": "{{.Name}}"} in /etc/docker/daemon.json

Now i can use if "xxx" in [xxxx][xxx] or if [xxxx][xxxx][xxx] =~ "filebeat"just like following:

if [docker][attrs][tag] =~ "filebeat" {
        drop {}
    }

It's working good.

But,when i use filter plugin to add new fields in logstash. i can't use if [xxx][xxx] to fliter something.
for example:
I use grok match the nginx access log to get "%{QS:useragent}" and use useragent to get some fields.

useragent {
            source => "useragent"
            target => "client.agent"
        }

I got some filed like client.agent.name client.agent.device and more.

Now when i set filter if [client][agent][device] =~ "Spider" { drop {} } it doesn't work.
I don't kown why....

Another question about fields syntax:
What is the difference between [docker][attrs][tag] and docker.attrs.tag ? sometime docker.attrs.tag is worked sometime does not worked.

[Badger]

@stille

target => "client.agent"

That creates fields inside the object [client.agent]. So you have fields like [client.agent][device]. Most likely you want to use

target => "[client][agent]"

[stille]

Thanks for help
It's working now.

So about another question mayby i knew it.
clients.agentIn logstash also is [clients.agent]will be show clients.agenton kibana
[client][agent] in logstash will be show clients.agenton kibana too.
Turns out i confuse the two...
It's correct?

[cknz (Cameron Kerr)]

You may be less confused if you view the data in Kibana (in the Discover app) as JSON; this will strip away some of the presentation layer that Kibana will be doing.

[client][agent] = "foo" in Logstash will be the same as the following in JSON

{"client":
    {"agent": "foo"}
}

Kibana will display this as client.agent, which is why field names mustn't (shouldn't?) have dots in them.

I'm not certain what Logstash would make of the [client.agent][device] = "Spider", but I imagine it would probably result in the following JSON and should be avoided

{"client.agent":
    {"device": "Spider"}
}

On a similar point of confusion, note that Kibana will use '-' to indicate a null field. This is not the same as when the likes of Apache httpd puts a '-' in its access log to indicate a null field; in that case, without processing to strip it out, it will be exactly the string '-'. This is something that viewing records as JSON can make clearer.

[Badger]

@cknz

I'm not certain what Logstash would make of the [client.agent][device] = "Spider" , but I imagine it would probably result in the following JSON and should be avoided

{"client.agent":
    {"device": "Spider"}
}

I am certain and that is exactly what it will do. logstash has no problem with periods in object or field names because its syntax is not ambiguous.

elasticsearch originally allowed it, then banned it, then patched the parser to provide error messages that made the conflict between the two clearer when you tried to use both, so it reënabled it.

Periods in field and object names are a terrible idea, but the product allows you to implement bad ideas.

[stille]

@cknz @Badger

Thanks both of you again .

I used the period as a subfield delimite.

I checked oldest logs as JSON , you're right.client.agentis a field, is not a subfield.

Just like[client][agent] is the best solution.