Parsing Mapping and finding the right ECS fields for logs in general

[cgekoski (Cory)]

Hello everyone. I'm going to start with saying sorry TLDR but hope someone can shine the light in the right direction. Pretty new to logstash/elastic and trying to come to grasp with the ECS field guide and want to teach myself to map fields correctly, mostly so that the elastic SIEM works or other features that are built-in to enterprise elastic, etc. works out of the box. Where i struggle is what to use for the fields as i haven't really found anything written in stone. For example in the below log (IP's redacted)

2020-12-11T11:52:36-05:00 ns3 client 134.192.2.1#59359 (pagead.l.doubleclick.net): view Internal: query: pagead.l.doubleclick.net IN AAAA +ED (134.192.1.1)

Starts simple with;

%{TIMESTAMP_ISO8601} %{WORD:???}

The second field "ns3" is the host name of the dns server. should i be mapping it as host.name, host.hostname, observer.hostname, something i make up but consistent in my environment, etc?.

The third field i can just add to my grok statement "client" and roll with the next which is the source ip of the query, or it could be considered the client.

%{TIMESTAMP_ISO8601} %{WORD:???} client %{IP:???}

Same scenario the IP could be mapped as multiple fields like; client.ip, client.address, source.address, source.ip, dns.answers.data, Local.Address, etc.

%{TIMESTAMP_ISO8601} %{WORD:} client %{IP}#%{INT:}

I dont think the integer is valuable for mapping but again how do you determine a good map for it?

%{TIMESTAMP_ISO8601} %{WORD:} client %{IP}#%{INT:} (%{NOTSPACE:???})

The next one i thought would be easy "(pagead.i.doubleclick.net)" in the dns section but wrong again. it could be dns.answers.name, dns.question.name. Im thinking this should be the dns.question though as its followed by query?

For the next three i think its straight forward;
"IN" equals dns.question.class
"AAAA" equals dns.question.type
"+ED" equals dns.header_flags

The final piece to this gem and i really appreciate you getting this far, is an IP address which is the host ip of the dns server. So should this be observer.ip, source.address, source.ip, host.ip, etc?

Maybe im over thinking this or making it more difficult than it should be?

Thanks for any feedback in advance,

Cory

[ebeahan (Eric Beahan)]

Hi @cgekoski - welcome to the community!

A few general resources I'd point you towards to help get started are the Elastic Security ECS field reference, ECS Getting Started, and ECS best practices for mapping network events. These documents should give you a better idea of 1) what mappings to priority for the Security Solution and 2) some examples of ECS best practices.

I'll try to touch on your questions one-by-one.

@cgekoski

The second field "ns3" is the host name of the dns server. should i be mapping it as host.name, host.hostname, observer.hostname, something i make up but consistent in my environment, etc?.

If ns3 is the host this log was generated on, use host.name.

@cgekoski

Same scenario the IP could be mapped as multiple fields like; client.ip, client.address, source.address, source.ip, dns.answers.data, Local.Address, etc.

If this is the DNS client's IP querying the server, use client.ip.

@cgekoski

I dont think the integer is valuable for mapping but again how do you determine a good map for it?

I can't say for sure since I'm not sure what the data source is here. The value could be the client's source port. If so, there's client.port.

@cgekoski

The next one i thought would be easy "(pagead.i.doubleclick.net)" in the dns section but wrong again. it could be dns.answers.name, dns.question.name. Im thinking this should be the dns.question though as its followed by query?

Again, I can't say for sure without knowledge about the source, but if we assume it's the queried name, use dns.question.name.

@cgekoski

For the next three i think its straight forward;
"IN" equals dns.question.class
"AAAA" equals dns.question.type
"+ED" equals dns.header_flags

The first two I agree with, but the third one is interesting. The value isn't listed in the docs for dns.header_flags as expected. It may just be a vendor's variant of DO.

@cgekoski

The final piece to this gem and i really appreciate you getting this far, is an IP address which is the host ip of the dns server. So should this be observer.ip, source.address, source.ip, host.ip, etc?

If the DNS server responds to the client's queries here, I'd suggest server.ip for the DNS server's IP address.