Convert fields to ECS

Elastic Stack Logstash
1

Hello,

I´m trying to ingest log file using ECS (Elastic Common Schema). For doing it, I have to rename source fields following the ECS convention. I made a filter which is not working:

filter {
json {
source => "message"
}

mutate {
rename => { "EventType" => "dns.type" }
rename => { "Timestamp" => "@timestamp" }
rename => { "ResponseCode" => "dns.repose_code" }
rename => { "InternalIp" => "source.ip" }
rename => { "Action" => "event.action" }
rename => { "Identities" => "event.module" }
rename => { "Domain" => "dns.question.name" }
rename => { "Categories" => "dns.answers.type" }
rename => { "QueryType" => "dns.op_code" }
rename => { "sourceFile" => "file.name" }
remove_field => [ "ExternalIp","MostGranularIdentity" ]
}
date {
match => ["@timestamp", "yyyy-MM-dd HH:mm:ss"]
}
}

I don´t know why it´s not working. I´m using rubydebug output and no data is showed in console.
While I was using that filter without mutate part (only json used), it was properly working, but obviusly ingested fields were not properly named following ECS.

The source log file is a json like this one:
{"sourceFile":"dns.csv.gz","EventType":"DNSLog","Timestamp":"2020-03-03 20:41:31","MostGranularIdentity":"TRYINGDNS","Identities":"TRYINGDNS","InternalIp":"10.10.10.10","ExternalIp":"10.10.10.10","Action":"Allowed","QueryType":"1 (C)","ResponseCode":"NOERROR","Domain":"www.google.com","Categories":"Software/Technology,Business Services"}

Am I doing it properly? Why it´s not working?
I also tryed without using the "remove_field". Also does not work.

Thanks.

Greetings.

2

ECS objects are objects, so in logstash you would refer to dns.question.name as [dns][question][name].

We have no way of telling why your input might not be generating events if you do not show us the input.

3

This is my Input:

input {
file {
path => "/home/user/Downloads/file.log"
start_position => "beginning"
}
}

So you mean I have to rename fields in that way?:
rename => { "Domain" => [dns][question][name] }

I was following the guide:
https://www.elastic.co/guide/en/ecs/current/ecs-dns.html

And I just decided which fields can correspond with ECS fields... I thought that renaming fields was the way to get ECS convention, I mean, mapping the fields to ECS fields so follow ECS convention.

4

I mean that you should use

rename => {
    "EventType" => "[dns][type]"
    "ResponseCode" => "[dns][response_code]"
    "InternalIp" => "[source][ip]"
    ...
}

If you have a "dns" object that contains a type field then in elasticsearch that would be called dns.type. In logstash it is referred to as [dns][type]

5

I just changed it following your example:
mutate {
rename => { "EventType" => "[dns][type]" }
rename => { "ResponseCode" => "[dns][repose_code]" }
rename => { "InternalIp" => "[source][ip]" }
rename => { "Action" => "[event][action]" }
rename => { "Identities" => "[event][module]" }
rename => { "Domain" => "[dns][question][name]" }
rename => { "Categories" => "[dns][answers][type]" }
rename => { "QueryType" => "[dns][op_code]" }
rename => { "sourceFile" => "[file][name]" }
#remove_field => [ "ExternalIp","MostGranularIdentity" ]
}
}

It´s not working... no data showed in console. The source log with input values are in first message I sent in this thread.

6

Are you appending lines to the file? If logstash has already read it then it will not re-read it.

7

Oh! you are right. I tryed adding lines and it worked.
How can I do to rename the field Timestamp to ECS? what should be the ECS field name? Is Timestamp valid?

Thanks

8

I know very little about ECS, so I cannot answer that.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.