# Filebaet and logstash encoding problem

**URL:** <https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [April 18, 2023, 1:10pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223 "2023-04-18T13:10:13Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 18, 2023, 1:10pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/1 "2023-04-18T13:10:13Z")

</div>

Hello, I can't understand why Logstash doesn't analyze logs from filebeat correctly, I see strange errors like:

```auto
JSON parsing error, source data now in message field {:message=>"Unexpected character ('*' (code 42)): expected valid value (number, string, array, object, 'true', 'false' or 'null')\n at [Source: (String)\"*\\\\u0019 6y~!\\\xDC$V........

```

```auto
Received event with character encoding other than configured. {:text=>"l\\x8E\\xED\\u0003\\t\\xCC\\u0014\\xC3", :expected_charset=>"UTF-8"}

```

My logstash config:

```auto
input {
  tcp {
    port => 5046
    codec => json{charset => "ISO-8859-1"}
    ssl => false
  }
}

filter {
  json {
    source => "message"
    target => "json"
    remove_field => ["message"]
  }
}

output {
  elasticsearch { 
    hosts => ["192.168.1.14:9200"] 
    index => "logstash-%{[@metadata][indexDate]}"
  }
  stdout { codec => rubydebug }
}

```

My Filebeat config:

```auto
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

processors:
  - add_docker_metadata:
      host: 'unix:///var/run/docker.sock'

  - decode_json_fields:
      fields: ["message"]
      target: "json"
      overwrite_keys: true

output.logstash:
  hosts: ["logstash:5046"].
  protocol: "tcp

logging.json: true
logging.metrics.enabled: false

```

I tried to change the parameters of logstash, but it did not give me any results, version filebeat 7.2.0, logstash 7.17.9, I also tried to run containers on a newer version 8.7.0, but the problem remains, I have looked through many forums, but I have not found a typical solution.

```auto
charset => "ISO-8859-1 and charset utf-8

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 18, 2023, 1:29pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/2 "2023-04-18T13:29:41Z")

</div>

You are using the wrong input in logstash, if you are sending logs with any beat you should use the `beats` input.

Change your input to this:

```auto
input {
  beats {
    port => 5046
  }
}

```

Also, in your `filebeat.yml` remove the `protocol` from your logstash output, this setting does not exist.

It should be just:

```auto
output.logstash:
  hosts: ["logstash:5046"]

```

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 18, 2023, 1:54pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/3 "2023-04-18T13:54:26Z")

</div>

Thanks for your help, I edited the config like you said, but I still see these errors when looking at the container logs logstash

```auto
Received an event that has a different character encoding than you configured. {:text=>"C\\u0014tʄ\\u0016\\x9A \\x....

```

```auto
JSON parse error, original data now in message field {:message=>"Unrecognized token 'h8': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"h8%\

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 18, 2023, 2:24pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/4 "2023-04-18T14:24:37Z")

</div>

Please share the entire log error, not just part of it.

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 18, 2023, 3:02pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/5 "2023-04-18T15:02:08Z")

</div>

Uploaded to google drive:

> **[logstash.txt](https://drive.google.com/file/d/1piF6-1FeXZYhTsXRyLiZjFZoisV5mZcI/view?usp=share_link)**
>
> Google Drive file.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 18, 2023, 3:04pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/6 "2023-04-18T15:04:31Z")

</div>

This file is too big, more them 700 MB.

You need to share only the relevant log message, not the entire log file, look at the message where you have the error and share that entire message.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [April 18, 2023, 3:19pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/7 "2023-04-18T15:19:27Z")

</div>

maks1001281 just copy few lines inside \< /\>

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 18, 2023, 3:20pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/8 "2023-04-18T15:20:59Z")

</div>

Set the beginning of loading and informative recurring events

> **[Log.txt](https://drive.google.com/file/d/1XsjAa4cRoNNg4nCtA286bi-AJ3dSdhPP/view?usp=share_link)**
>
> Google Drive file.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 18, 2023, 3:40pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/9 "2023-04-18T15:40:44Z")

</div>

Did you change your input to `beats`? The log you shared says that it is still using the tcp input.

```auto
[2023-04-18T12:00:09,076][INFO][logstash.inputs.tcp][main] Automatically switching from json to json_lines codec {:plugin=>"tcp"}
[2023-04-18T12:00:09,101][WARN][deprecation.logstash.codecs.jsonlines][main] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-04-18T12:00:09,225][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"}
[2023-04-18T12:00:09,245][INFO][logstash.inputs.tcp][main][af546bd34ee6b3498c79240da7941004db5763a57a03ba2f4588b4f793cac683] Starting tcp input listener {:address=>"0.0.0.0:5046", :ssl_enable=>false}

```

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 18, 2023, 4:10pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/10 "2023-04-18T16:10:57Z")

</div>

Yes, my current config:

Filebeeat:

```auto
filebeat.inputs:
  - type: container
    paths:
      - `` /var/lib/docker/containers/*/*.log''

processors:
  - add_docker_metadata:
      host: 'unix:///var/run/docker.sock'

  - decode_json_fields:
      fields: ["message"]
      target: "json"
      overwrite_keys: true

output.logstash:
  hosts: ["192.168.1.14:5046"]

logging.json: true
logging.metrics.enabled: false

```

Logstash:

```auto
input {
  beats {
    port => 5046
    codec => json{charset => "ISO-8859-1"}
  }
}

filter {
  json {
    source => "message"
    target => "json"
    remove_field => ["message"]
  }
}

output {
  elasticsearch { 
    hosts => ["192.168.1.14:9200"] 
    index => "logstash-%{[@metadata][indexdate]}"
  }
  stdout { codec => rubydebug }
}

```

I've now reread the log for 10 times and noticed the error:  
"reason"=\>"Invalid index name [logstash-%{[@metadata][indexDate]}], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"logstash-%{[@metadata][indexDate]}"}}}}

I corrected the index and everything worked! the index appeared in Kibane!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 18, 2023, 5:04pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/11 "2023-04-18T17:04:12Z")

</div>

> [@maks1001281](#):
>
> `codec => json{charset => "ISO-8859-1"}`

Remove this, I don't think this is needed, the beats output to logstash using `utf-8` if I'm not wrong.

> [@maks1001281](#):
>
> ```auto
> json {
> source => "message"
> target => "json"
> remove_field => ["message"]
> }
> 
> ```

I don't think you need this, you are already parsing the json message in your `filebeat.yml` with the `decode_json_fields` processor.

> [@maks1001281](#):
>
> "reason"=\>"Invalid index name [logstash-%{[@metadata][indexDate]}], must be lowercase", "index\_uuid"=\>"_na_", "index"=\>"logstash-%{[@metadata][indexDate]}"}}}}

This happens because the field `[@metadata][indexDate]` does not exist in some document, so it will use this literal value, which will give you the error since it has a uppercase letter.

Where is this field comming from? There is nothing in your logstash pipeline that creates it.

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 18, 2023, 6:34pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/12 "2023-04-18T18:34:18Z")

</div>

I deleted the config

```auto
{charset => "ISO-8859-1"}

```

I deleted this from the config and logstash didn't run because of errors, I put it back the way it was

```auto
json {
    source => "message"
    target => "json"
    remove_field => ["message"]
  }

```

I wasn't happy for a long time, logstash looped and started clogging the disk space fast and I don't know why, in kibane I see an error

```auto
index [.async-search] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];

```

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 19, 2023, 7:49am UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/13 "2023-04-19T07:49:59Z")

</div>

In the logs I see a parsing error and infinitely repeated values

```auto
[2023-04-19T13:54:33,530][WARN][logstash.filters.json][main][e9badb283d4469c6c8ba51c6d06909a513fde1fa93cd91e27af053107ad47e60] Error parsing json {:source=>"message", :raw=>"[2023-04-19T13:54:26.943+00:00][INFO][plugins-service] Plugin \"profiling\" is disabled.", :exception=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): was expecting comma to separate Array entries
 at [Source: (byte[])"[2023-04-19T13:54:26.943+00:00][INFO][plugins-service] Plugin "profiling" is disabled."; line: 1, column: 7]>}

```

```auto
[2023-04-19T13:54:36,338][WARN][logstash.filters.json][main][e9badb283d4469c6c8ba51c6d06909a513fde1fa93cd91e27af053107ad47e60] Error parsing json {:source=>"message", :raw=>" \"org_label-schema_url\" => \"https://www.elastic.co/products/beats/filebeat\"", :exception=>#<LogStash::Json::ParserError: Unexpected character ('=' (code 61)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])" "org_label-schema_url" => "https://www.elastic.co/products/beats/filebeat""; line: 1, column: 56]>}

```

```auto
{
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
      "@version" => "1",
        "stream" => "stderr",
    "@timestamp" => 2023-04-19T13:54:07.595Z,
         "agent" => {
            "hostname" => "5e4a6f707fd2",
        "ephemeral_id" => "e20cbf65-fa84-497d-aeca-f159d5cd69c2",
                  "id" => "a0cda17a-2e91-4dc7-a037-a94498c490d3",
                "type" => "filebeat",
             "version" => "7.2.0"
    },
          "host" => {
        "name" => "5e4a6f707fd2"
    },
           "ecs" => {
        "version" => "1.0.0"
    },
     "container" => {
            "id" => "5e4a6f707fd2c863360c122b45e48a066ca33d9cbd744acd46c7d38db35bf56c",
          "name" => "filebeat",
        "labels" => {
                           "org_label-schema_version" => "7.2.0",
                                        "description" => "Filebeat sends log files to Logstash or directly to Elasticsearch.",
                           "org_label-schema_license" => "GPLv2",
                "com_docker_compose_container-number" => "1",
             "com_docker_compose_project_working_dir" => "/home/maks/docker/ELK/help",
                           "org_label-schema_vcs-url" => "github.com/elastic/beats",
                    "org_label-schema_schema-version" => "1.0",
                      "com_docker_compose_depends_on" => "logstash:service_started:false",
                         "com_docker_compose_service" => "filebeat",
                          "com_docker_compose_oneoff" => "False",
            "com_docker_compose_project_config_files" => "/home/maks/docker/ELK/help/docker-compose.yml",
                         "com_docker_compose_version" => "2.17.2",
                           "org_label-schema_vcs-ref" => "9ba65d864ca37cd32c25b980dbb4020975288fc0",
                                            "license" => "Elastic License",
                        "org_label-schema_build-date" => "20190305",
                              "org_label-schema_name" => "filebeat",
                            "org_label-schema_vendor" => "Elastic",
                     "com_docker_compose_config-hash" => "d4907d7bd52514db27464e7319fd7e4f0fe65d53a162b59054f7edf571395348",
                           "com_docker_compose_image" => "sha256:30ee5f4f93abd3d63181ea1706f54a1c14a61045f5a2a6a4bbf1562b13937d9a",
                               "org_label-schema_url" => "https://www.elastic.co/products/beats/filebeat",
                         "com_docker_compose_project" => "help"
        },
         "image" => {
            "name" => "elastic/filebeat:7.2.0"
        }
    },
          "json" => {
            "level" => "info",
        "timestamp" => "2023-04-19T13:54:07.595Z",
           "caller" => "instance/beat.go:292",
          "message" => "Setup Beat: filebeat; Version: 7.2.0"
    },
         "input" => {
        "type" => "container"
    },
           "log" => {
          "file" => {
            "path" => "/var/lib/docker/containers/5e4a6f707fd2c863360c122b45e48a066ca33d9cbd744acd46c7d38db35bf56c/5e4a6f707fd2c863360c122b45e48a066ca33d9cbd744acd46c7d38db35bf56c-json.log"
        },
        "offset" => 4845
    }
}

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [April 19, 2023, 8:13am UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/14 "2023-04-19T08:13:23Z")

</div>

> [@maks1001281](#):
>
> `disk usage exceeded flood-stage watermark`

Do you have enough disk space? Do now let the free disk space bellow 15%. Check [this](https://www.elastic.co/guide/en/elasticsearch/reference/master/fix-watermark-errors.html)

---

<div class="post-metadata">

**Author:** ![maks1001281](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maks1001281](https://discuss.elastic.co/u/maks1001281)\
**Post date:** [April 19, 2023, 1:27pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/15 "2023-04-19T13:27:01Z")

</div>

I set the Elasticsearch config:  
cluster.routing.allocation.disk.watermark.flood\_stage  
But this did not help me, after deleting the container and raising a new one, the disk rapidly begins to run out of space, 30 gb in 15 minutes, and all this space is taken by the volume logstash

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [April 19, 2023, 3:00pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/16 "2023-04-19T15:00:01Z")

</div>

I would avoid debug mode, comment out: `#stdout { codec => rubydebug }`  
Are you sure LS consumes a lot of space?  
Are you using in-memory queue processing or persistant on disk?  
Are you set `log.level: info`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 17, 2023, 3:00pm UTC](https://discuss.elastic.co/t/filebaet-and-logstash-encoding-problem/330223/17 "2023-05-17T15:00:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.