# Docker logs includes unreadable in Kibana

**URL:** <https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [April 25, 2022, 7:21pm UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196 "2022-04-25T19:21:02Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![dev9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dev9/32/81258_2.png) [@dev9](https://discuss.elastic.co/u/dev9)\
**Post date:** [April 25, 2022, 7:21pm UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196/1 "2022-04-25T19:21:02Z")

</div>

Hi,  
I am using filebeat version 7.17.3 running on Ubuntu 18.04.6 LTS to ship docker logs using filebeat installation on the host OS. I am not using a filebeat container.  
I noticed a couple of things that I would like to remediate and I couldn't find an answer for in the documentation.  
My .yml file is

```auto
setup.ilm.enabled: auto
setup.template.name: "myindex"
setup.template.pattern: "myindex-*"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.rollover_alias: "myindex"
# =======================Containers Settings=================================

filebeat.autodiscover:
  providers:
      - type: docker
        hints.enabled: true
        hints.default_config:
          type: container
          paths:
            - /var/lib/docker/*/containers/*/*.log
        exclude_lines: ["^\\s+[\\-`('.|_]"]
        json.key_under_root: true
        json.ignore_decoding_error: true  
        json.add_error_key: true
        json.message_key: log

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /home/username/documents/projects/filebeat/*.log
processors:
  - add_docker_metadata: ~

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.kibana:
output.elasticsearch:
  hosts: ["REMOVED"]
  index: "unity-%{+yyyy.MM.dd}"
  ssl.certificate_authorities: ["REMOVED"]
processors:
- decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 2
      target: ""
      overwrite_keys: true
- rename:
      fields:
        - from: "source"
          to: "msource"
      ignore_missing: true
      fail_on_error: false

cloud.id: "${ES_USER}" 
cloud.auth: "${ES_PASS}" 

```

While the logs are shipped to Kibana, they are not readable as they include the terminal color codes like `“ [1;34mbin[m [1;34mctmp[m [1;34mdev[m [1;34metc[m [1;34mhome[m [1;34mlib[m [1;34mmedia[m [1;34mmnt[m [1;34mopt[m [1;34mproc[m [1;34mroot[m [1;34mrun[m [1;34msbin[m [1;34msrv[m [1;34msys[m [1;34mtmp[m [1;34musr[m [1;34mvar[m”` is what I see when I run ls -la inside a container.

How can I only show the text?  
I also don't see any logs regarding to containers starting or stopping. Is this an expected behaviour?

Thanks

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [April 26, 2022, 6:29am UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196/2 "2022-04-26T06:29:15Z")

</div>

Hello @dev9

> [@](#):
>
> How can I only show the text?

Indeed this is the actual content of the log entry, you either have to disable coloured output in the docker container, or add a `script` processor ([Script Processor | Filebeat Reference [7.17] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.17/processor-script.html)) in order to strip the control chars

I found a regex that seems to do the work at [How to remove ^[, and all of the ANSI escape sequences in a file using linux shell scripting - Stack Overflow](https://stackoverflow.com/questions/6534556/how-to-remove-and-all-of-the-escape-sequences-in-a-file-using-linux-shell-sc/59043663#59043663)

```auto
printf '\e[31m%s\e[0m' "this is in red"|sed 's/\x1B\[[0-9;]*[JKmsu]//g'
```

You have to port it to a `script` processor

---

<div class="post-metadata">

**Author:** ![dev9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dev9/32/81258_2.png) [@dev9](https://discuss.elastic.co/u/dev9)\
**Post date:** [April 26, 2022, 12:21pm UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196/3 "2022-04-26T12:21:02Z")

</div>

Hi Andrea,  
Is this the only option I have to sanitize the logs even though I am using the official provider?  
Any idea why I only see logs if I exec inside a container? or How to reduce the number of documents added to our index. For example, 24 records get added if I do ls -la inside a container because filebeat sends every line as its own record.

---

<div class="post-metadata">

**Author:** ![dev9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dev9/32/81258_2.png) [@dev9](https://discuss.elastic.co/u/dev9)\
**Post date:** [April 26, 2022, 5:58pm UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196/4 "2022-04-26T17:58:43Z")

</div>

I managed to remove the ascii characters and cleanup the kibana entries using the processor script. In case anyone else has the same issue. the solution is

```auto
setup.ilm.enabled: auto
setup.template.name: "myindex"
setup.template.pattern: "myindex-*"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.rollover_alias: "myindex_ILM"
# =======================Containers Settings=================================

filebeat.autodiscover:
  providers:
      - type: docker
        hints.enabled: true
        hints.default_config:
          type: container
          paths:
            - /var/lib/docker/*/containers/*/*.log
        exclude_lines: ["^\\s+[\\-`('.|_]"]
        json.key_under_root: true
        json.ignore_decoding_error: true  
        json.add_error_key: true
        json.message_key: log

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /home/USER/documents/projects/filebeat/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.kibana:
output.elasticsearch:
  hosts: ["REMOVED"]
  index: "myindex-%{+yyyy.MM.dd}"
  ssl.certificate_authorities: [REMOVED]
processors:
- decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 2
      target: ""
      overwrite_keys: true
- rename:
      fields:
        - from: "source"
          to: "msource"
      ignore_missing: true
      fail_on_error: false
- script:
    lang: javascript
    source: >
      function process(event){
        var regex = new RegExp('\x1B(?:[@-Z\\-_]|\[[0-?]*[-/]*[@-~])','g');
        var clean = event.Get('message');
        clean = clean.replace(regex, '');
        event.Put('message',clean);    
        return event;
      }
    
cloud.id: "${ES_USER}" 
cloud.auth: "${ES_PASS}" 

```

I still however only see events logged if I exec inside the container. None of the container states is logged.

---

<div class="post-metadata">

**Author:** ![Andrea\_Spacca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_spacca/32/76976_2.png) [@Andrea\_Spacca](https://discuss.elastic.co/u/Andrea_Spacca)\
**Post date:** [April 27, 2022, 1:56am UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196/5 "2022-04-27T01:56:55Z")

</div>

Hello @dev9 ,

filebeat will collects logs generated by the containers, not the state of the containers

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 25, 2022, 3:57am UTC](https://discuss.elastic.co/t/docker-logs-includes-unreadable-in-kibana/303196/6 "2022-05-25T03:57:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.