# Logstash - Manipulate field with special characters

**URL:** <https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203>\
**Category:** Logstash\
**Created:** [November 2, 2018, 4:22pm UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203 "2018-11-02T16:22:35Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 2, 2018, 4:22pm UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/1 "2018-11-02T16:22:35Z")

</div>

Hello,

I'm trying to manipulate a bunch of fields that have names with special characters (e.g. @ or dots).  
This is a usual case if someone uses [Logstash-logback-encoder](https://github.com/logstash/logstash-logback-encoder) for shipping logs from an application to logstash.

Unfortunately, neither mutate nor ruby seems to do the trick.  
Snippets of codes that I have tried:

```
if [@fields.request_headers.user-agent] =~ "ndroid" {
   mutate {
           add_field => { "android" => "true" }
   }
   mutate {
           remove_field => ["@fields.request_headers.user-agent"]
   }	   
}

```

or

```
def filter(event)
  unless event.get("@fields.request_headers.user-agent").nil?
    if event.get("@fields.request_headers.user-agent").include? "ndroid"
      event.set('andoid', true)
      event.remove('@fields.request_headers.user-agent')
    end
 end
 return [event]
end

```

In the past I have successfully utilized both methods for fields having proper names.  
So, I was wondering if this is a known issue for logstash or ruby filter.

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 8:01am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/2 "2018-11-30T08:01:07Z")

</div>

This must be a common problem since several frameworks utilize this naming pattern (e.g. Elastic APM).  
Any ideas?

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 8:10am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/3 "2018-11-30T08:10:15Z")

</div>

Do you have any error messages or what are the results you are getting vs what you expect?

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 9:00am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/4 "2018-11-30T09:00:57Z")

</div>

Hello @Eniqmatic!

There are no errors during the start-up of Logstash nor during its execution.  
It's just that the mutation actions are not executed.

Let me provide two more examples:

_ **Example 1:** _

```
if [@fields.request_headers.user-agent] =~ "ndroid" {
    mutate {
           add_field => { "android" => "true" }
   }
}
else {
    mutate {
           remove_field => ["@fields.request_headers.user-agent"]
   }	   
}

```

The aforementioned example is not working as no mutation action is being executed.

_ **Example 2:** _

```
if [@fields.request_headers.user-agent] =~ "ndroid" {
    mutate {
           add_field => { "android" => "true" }
   }
}
else {
   drop {}	   
}

```

The aforementioned example makes all messages to drop.

It seems that Logstash has trouble evaluating any type of expression (either at branch level or at mutation level) that contains field names with special characters (such as dots).

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 9:03am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/5 "2018-11-30T09:03:29Z")

</div>

OK that makes better sense. So your if statement is not correct. Can you show me your field names please?

I won't know for sure till I see your fields but I think you need this instead:

```
if [request_headers][user-agent] =~ "ndroid" {
```

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 9:10am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/6 "2018-11-30T09:10:13Z")

</div>

I believe the if statement is correct since there is a field with name: `@fields.request_headers.user-agent` and it contains "`ndroid`".  
I can see it both via the Search API of Elasticsearch and the Discover tab of Kibana (see the screenshot: [https://imgur.com/a/pkw56EB](https://imgur.com/a/pkw56EB)).

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 9:12am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/7 "2018-11-30T09:12:30Z")

</div>

OK, but the request should still be in square brackets for nested fields:

```
if [@fields][request_headers][user-agent] =~ "ndroid" {
```

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 9:16am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/8 "2018-11-30T09:16:10Z")

</div>

I'll try that and revert with an update.  
However, please not that this is not a typical case of nested fields; it's a case of a field name following a bad practice when it comes to naming conventions.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 9:17am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/9 "2018-11-30T09:17:51Z")

</div>

I agree, I wouldn't typically use "@fields" in the field name, request\_headers.user-agent is a better name!

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 9:29am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/10 "2018-11-30T09:29:45Z")

</div>

Update: unfortunately, traversing using square brackets did not work either.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 9:34am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/11 "2018-11-30T09:34:49Z")

</div>

Can you try:

```
([@fields][request_headers][user-agent] =~ "ndroid")

```

Can you also show me the raw JSON document in kibana?

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 9:44am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/12 "2018-11-30T09:44:12Z")

</div>

Using parentheses did not help either. 😕  
This is the raw JSON document: [https://jsonblob.com/9dd7d201-f483-11e8-a31c-2b4675e319d9](https://jsonblob.com/9dd7d201-f483-11e8-a31c-2b4675e319d9)

This works: `if [@fields.request_headers][user-agent] =~ "ndroid"`

Good idea asking for the raw JSON doc!

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 9:53am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/13 "2018-11-30T09:53:26Z")

</div>

Strange, I have the exact same in my own config which works.

Another method you could try is:

```
if "ndroid" in [@fields][request_headers][user-agent] {
```

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 9:54am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/14 "2018-11-30T09:54:52Z")

</div>

I'm using Logstash 5.6.9  
What's your version?

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 30, 2018, 9:55am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/15 "2018-11-30T09:55:45Z")

</div>

I use 6.5 currently but have upgraded from 5.5 and use it there also!

---

<div class="post-metadata">

**Author:** ![g.le](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/g.le/32/22526_2.png) [@g.le](https://discuss.elastic.co/u/g.le)\
**Post date:** [November 30, 2018, 10:26am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/16 "2018-11-30T10:26:08Z")

</div>

Thanks again @Eniqmatic!  
We would never reach the solution had you not suggested taking a closer look to the raw JSON document.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2018, 10:26am UTC](https://discuss.elastic.co/t/logstash-manipulate-field-with-special-characters/155203/17 "2018-12-28T10:26:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.