Logstash - Manipulate field with special characters

[g.le]

Hello,

I'm trying to manipulate a bunch of fields that have names with special characters (e.g. @ or dots).
This is a usual case if someone uses Logstash-logback-encoder for shipping logs from an application to logstash.

Unfortunately, neither mutate nor ruby seems to do the trick.
Snippets of codes that I have tried:

if [@fields.request_headers.user-agent] =~ "ndroid" {
   mutate {
           add_field => { "android" => "true" }
   }
   mutate {
           remove_field => [ "@fields.request_headers.user-agent" ]
   }	   
}

or

def filter(event)
  unless event.get("@fields.request_headers.user-agent").nil?
    if event.get("@fields.request_headers.user-agent").include? "ndroid"
      event.set('andoid', true)
      event.remove('@fields.request_headers.user-agent')
    end
 end
 return [event]
end

In the past I have successfully utilized both methods for fields having proper names.
So, I was wondering if this is a known issue for logstash or ruby filter.

[g.le]

This must be a common problem since several frameworks utilize this naming pattern (e.g. Elastic APM).
Any ideas?

[Eniqmatic (Lewis Barclay)]

Do you have any error messages or what are the results you are getting vs what you expect?

[g.le]

Hello @Eniqmatic!

There are no errors during the start-up of Logstash nor during its execution.
It's just that the mutation actions are not executed.

Let me provide two more examples:

Example 1:

if [@fields.request_headers.user-agent] =~ "ndroid" {
    mutate {
           add_field => { "android" => "true" }
   }
}
else {
    mutate {
           remove_field => [ "@fields.request_headers.user-agent" ]
   }	   
}

The aforementioned example is not working as no mutation action is being executed.

Example 2:

if [@fields.request_headers.user-agent] =~ "ndroid" {
    mutate {
           add_field => { "android" => "true" }
   }
}
else {
   drop {}	   
}

The aforementioned example makes all messages to drop.

It seems that Logstash has trouble evaluating any type of expression (either at branch level or at mutation level) that contains field names with special characters (such as dots).

[Eniqmatic (Lewis Barclay)]

OK that makes better sense. So your if statement is not correct. Can you show me your field names please?

I won't know for sure till I see your fields but I think you need this instead:

if [request_headers][user-agent] =~ "ndroid" {

[g.le]

I believe the if statement is correct since there is a field with name: @fields.request_headers.user-agent and it contains "ndroid".
I can see it both via the Search API of Elasticsearch and the Discover tab of Kibana (see the screenshot: https://imgur.com/a/pkw56EB).

[Eniqmatic (Lewis Barclay)]

OK, but the request should still be in square brackets for nested fields:

if [@fields][request_headers][user-agent] =~ "ndroid" {

[g.le]

I'll try that and revert with an update.
However, please not that this is not a typical case of nested fields; it's a case of a field name following a bad practice when it comes to naming conventions.

[Eniqmatic (Lewis Barclay)]

I agree, I wouldn't typically use "@fields" in the field name, request_headers.user-agent is a better name!

[g.le]

Update: unfortunately, traversing using square brackets did not work either.

[Eniqmatic (Lewis Barclay)]

Can you try:

([@fields][request_headers][user-agent] =~ "ndroid")

Can you also show me the raw JSON document in kibana?

[g.le]

Using parentheses did not help either.
This is the raw JSON document: https://jsonblob.com/9dd7d201-f483-11e8-a31c-2b4675e319d9

This works: if [@fields.request_headers][user-agent] =~ "ndroid"

Good idea asking for the raw JSON doc!

[Eniqmatic (Lewis Barclay)]

Strange, I have the exact same in my own config which works.

Another method you could try is:

if "ndroid" in [@fields][request_headers][user-agent] {

[g.le]

I'm using Logstash 5.6.9
What's your version?

[Eniqmatic (Lewis Barclay)]

I use 6.5 currently but have upgraded from 5.5 and use it there also!

[g.le]

Thanks again @Eniqmatic!
We would never reach the solution had you not suggested taking a closer look to the raw JSON document.