Translate Filter Destination

[empfangsfehler]

Hi there,

i would like to store my translated field as "dhcp.EventName" but it seems not to work, the field is never created or filled. I haven't found anything in the documentation about this.

I tried the following configurations:

....
if "dhcpservice" in [tags] {
	grok {
		match => { "message" => ['^%{NUMBER:dhcp.EventID},%{GREEDYDATA}'] }
	}
	mutate {
		convert => [ "[dhcp][EventID]", "integer"]
	}
	translate {
		field => "[dhcp][EventID]"
		destination => "[dhcp][EventName]"
		override => false
		dictionary_path => "/etc/logstash/dhcpdirectory.yml"
	}
}
....

or

....
if "dhcpservice" in [tags] {
	grok {
		match => { "message" => ['^%{NUMBER:dhcp.EventID},%{GREEDYDATA}'] }
	}
	mutate {
		convert => [ "[dhcp][EventID]", "integer"]
	}
	translate {
		field => "[dhcp][EventID]"
		destination => "dhcp.EventName"
		override => false
		dictionary_path => "/etc/logstash/dhcpdirectory.yml"
	}
}
....

and even

....
if "dhcpservice" in [tags] {
	grok {
		match => { "message" => ['^%{NUMBER:dhcp.EventID},%{GREEDYDATA}'] }
	}
	mutate {
		convert => [ "[dhcp][EventID]", "integer"]
		add_field => { "dhcp.EventName" => "Sampledata" }
	}
	translate {
		field => "[dhcp][EventID]"
		destination => "[dhcp][EventName]"
		override => true
		dictionary_path => "/etc/logstash/dhcpdirectory.yml"
	}
}
....

[Christian_Dahlqvist]

What does a sample event look like?

[empfangsfehler]

Find all three variants in the same order as in the original post on gist:

https://gist.github.com/empfangsfehler/11ae147d01cce05dc98c886364375c88

https://gist.github.com/empfangsfehler/5e1048e8104d2c8e7d35789c0669eb02

https://gist.github.com/empfangsfehler/bb6bb6c41ee247d6e7fd0e73bf8eda4c

Also the event which is read by Filebeat from a LogFile looks like that:

32,05/27/18,12:12:59,,172.16.50.230,LTK17024,,,0,6,,AAEBI7eicVFkGI8afQ0ILX0kB+R1WATYvcpJHhf0CihMicI=,

[rcowart (Rob Cowart)]

What does your dhcpdirectory.yml look like?

[empfangsfehler]

https://gist.github.com/empfangsfehler/8ed74df44d025bc6b132300a12cf091d

The Translate itself works fine when i use "dhcpEventName" as destination but not when i would like to save in a nested field.

[rcowart (Rob Cowart)]

A couple of things to consider...

1. You convert dhcp.EventID to an integer before the lookup. So the values 00, 01, 02 will likely not match as dhcp.EventID will contain 0, 1, 2. The values that are not 0-padded should be fine.

2. Add fallback => "UNKNOWN" to your translate filter. This will at least let you know whether the lookup is failing to find a match, or if the lookup is not even happening.

3. I would also recommend getting rid of the override setting unless the field and destination are the same field.

Here are a couple of examples that work fine for me...

translate {
  dictionary_path => "${ELASTIFLOW_DICT_PATH:/etc/logstash/elastiflow/dictionaries}/iana_protocol_numbers.yml"
  field => "[flow][ip_protocol]"
  destination => "[flow][ip_protocol]"
  fallback => "UNKNOWN(%{[flow][ip_protocol]})"
  override => true
}

translate {
  dictionary_path => "${ELASTIFLOW_DICT_PATH:/etc/logstash/elastiflow/dictionaries}/iana_service_names_udp.yml"
  field => "[flow][src_port]"
  destination => "[flow][src_port_name]"
  fallback => "__UNKNOWN"
}

[empfangsfehler]

Thanks!

I moved the integer convert after the translate.

The field is still not there - do you add this before the translate manually with mutate?

Here is my translate now:

translate {
	field => "[dhcp][EventID]"
	destination => "[dhcp][EventDescription]"
	dictionary_path => "/etc/logstash/dhcpdirectory.yml"
	fallback => "_UNKNOWN"
}

I tried also by setting the destination to dhcp.EventDescription

[rcowart (Rob Cowart)]

The only things that I can think of...

1. I never use dot notation to refer to nested fields. Try changing %{NUMBER:dhcp.EventID} to %{NUMBER:[dhcp][EventID]}. I realize that the grok debugger allows dot notation, but the grok debugger has a few inconsistencies with Logstash itself. The reason I wonder about this is that your sample data shows an underscore for the field name: "dhcp_EventID": 11,

2. If the above is not the issue, I question whether the "dhcpservice" in [tags] is actually matching.

[empfangsfehler]

Now it works fine - i changed the GROK pattern from dot notation to square brackets notation.

Thanks for all!