# Filebeat inserting extra Unicode characters

**URL:** <https://discuss.elastic.co/t/filebeat-inserting-extra-unicode-characters/371048>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 25, 2024, 11:33pm UTC](https://discuss.elastic.co/t/filebeat-inserting-extra-unicode-characters/371048 "2024-11-25T23:33:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\
**Post date:** [November 25, 2024, 11:33pm UTC](https://discuss.elastic.co/t/filebeat-inserting-extra-unicode-characters/371048/1 "2024-11-25T23:33:07Z")

</div>

This is the original entry in the log file:

```auto
2024-11-25 23:14:27,671 INFO o.s.w.s.c.WebSocketMessageBrokerStats [MessageBroker-1] WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 197]

```

I copied this file to my Windows machine and opened it in Notepad. The bottom of the Notepad window says the file is Unix (LF) UTF-8.

This is the same log record as shown in Kibana/Elasticsearch:

```auto
\u001b[30m2024-11-25 22:44:27,670\u001b[0;39m \u001b[34mINFO \u001b[0;39m [\u001b[34mMessageBroker-1\u001b[0;39m] \u001b[33mo.s.w.s.c.WebSocketMessageBrokerStats\u001b[0;39m: WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 196]\n

```

So somewhere in the process of Filebeat parsing the log file and shipping its content to Elasticsearch, to displaying the content in Kibana, something is inserting extra characters like `\u001b[30m` and `\u001b[34m`.

The applicable settings in our filebeat-kubernetes.yaml file

```auto
    - type: filestream
      id: ceo-api-dev1-container-logs
      paths:
        - /var/log/containers/ceo-api-*.log
      encoding: utf-8
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ceo
        data_stream.namespace: api
        app_id: ceo-api-dev1
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            namespace: ceo-dev1
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

```

According to the Filebeat docs, the `encoding: utf-8` setting should have told Filebeat to parse the log file as utf-8 characters.

Is there a way to prevent these extra characters from being added or are we stuck hacking with a Grok processor or adding Logstash to our setup to get rid of those extra characters?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 26, 2024, 3:52am UTC](https://discuss.elastic.co/t/filebeat-inserting-extra-unicode-characters/371048/2 "2024-11-26T03:52:48Z")

</div>

> [@paolovalladolid](#):
>
> So somewhere in the process of Filebeat parsing the log file and shipping its content to Elasticsearch, to displaying the content in Kibana, something is inserting extra characters like `\u001b[30m` and `\u001b[34m`.

Are you sure this isn't from the source?

What you shared are ANSI escape codes for colors, does your application generate logs in colors? In which language is it written? Some logging frameworks add colors by default.

Testing with an `echo -e` this is the result:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2f8318ff11d6dbcd33d37af3d267480c8b73c57f.jpeg)

I don't think that there is anything in the stack that would add those characters in that way.

---

<div class="post-metadata">

**Author:** ![paolovalladolid](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@paolovalladolid](https://discuss.elastic.co/u/paolovalladolid)\
**Post date:** [November 26, 2024, 10:18pm UTC](https://discuss.elastic.co/t/filebeat-inserting-extra-unicode-characters/371048/3 "2024-11-26T22:18:29Z")

</div>

Thanks!

It turns out the application had a logback-spring.xml file that was configured to use PatternLayout. The PatternLayout config had color strings which were then encoded as ANSI escape sequences.

Deleting the color strings solved the issue.