Filebaet and logstash encoding problem

Elastic Stack Logstash
1

Hello, I can't understand why Logstash doesn't analyze logs from filebeat correctly, I see strange errors like:

JSON parsing error, source data now in message field {:message=>"Unexpected character ('*' (code 42)): expected valid value (number, string, array, object, 'true', 'false' or 'null')\n at [Source: (String)\"*\\\\u0019 6y~!\\\xDC$V........

Received event with character encoding other than configured. {:text=>"l\\x8E\\xED\\u0003\\t\\xCC\\u0014\\xC3", :expected_charset=>"UTF-8"}

My logstash config:

input {
  tcp {
    port => 5046
    codec => json{charset => "ISO-8859-1"}
    ssl => false
  }
}

filter {
  json {
    source => "message"
    target => "json"
    remove_field => ["message"]
  }
}

output {
  elasticsearch { 
    hosts => ["192.168.1.14:9200"] 
    index => "logstash-%{[@metadata][indexDate]}"
  }
  stdout { codec => rubydebug }
}

My Filebeat config:

filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

processors:
  - add_docker_metadata:
      host: 'unix:///var/run/docker.sock'

  - decode_json_fields:
      fields: ["message"]
      target: "json"
      overwrite_keys: true

output.logstash:
  hosts: ["logstash:5046"].
  protocol: "tcp

logging.json: true
logging.metrics.enabled: false

I tried to change the parameters of logstash, but it did not give me any results, version filebeat 7.2.0, logstash 7.17.9, I also tried to run containers on a newer version 8.7.0, but the problem remains, I have looked through many forums, but I have not found a typical solution.

charset => "ISO-8859-1 and charset utf-8
2

You are using the wrong input in logstash, if you are sending logs with any beat you should use the beats input.

Change your input to this:

input {
  beats {
    port => 5046
  }
}

Also, in your filebeat.yml remove the protocol from your logstash output, this setting does not exist.

It should be just:

output.logstash:
  hosts: ["logstash:5046"]
1 Like
3

Thanks for your help, I edited the config like you said, but I still see these errors when looking at the container logs logstash

Received an event that has a different character encoding than you configured. {:text=>"C\\u0014tʄ\\u0016\\x9A \\x....

JSON parse error, original data now in message field {:message=>"Unrecognized token 'h8': was expecting ('true', 'false' or 'null')\n at [Source: (String)\"h8%\
4

Please share the entire log error, not just part of it.

5

Uploaded to google drive:

6

This file is too big, more them 700 MB.

You need to share only the relevant log message, not the entire log file, look at the message where you have the error and share that entire message.

7

maks1001281 just copy few lines inside < />

8

Set the beginning of loading and informative recurring events

9

Did you change your input to beats? The log you shared says that it is still using the tcp input.

[2023-04-18T12:00:09,076][INFO ][logstash.inputs.tcp      ][main] Automatically switching from json to json_lines codec {:plugin=>"tcp"}
[2023-04-18T12:00:09,101][WARN ][deprecation.logstash.codecs.jsonlines][main] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-04-18T12:00:09,225][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-04-18T12:00:09,245][INFO ][logstash.inputs.tcp      ][main][af546bd34ee6b3498c79240da7941004db5763a57a03ba2f4588b4f793cac683] Starting tcp input listener {:address=>"0.0.0.0:5046", :ssl_enable=>false}
10

Yes, my current config:

Filebeeat:

filebeat.inputs:
  - type: container
    paths:
      - `` /var/lib/docker/containers/*/*.log''

processors:
  - add_docker_metadata:
      host: 'unix:///var/run/docker.sock'

  - decode_json_fields:
      fields: ["message"]
      target: "json"
      overwrite_keys: true

output.logstash:
  hosts: ["192.168.1.14:5046"]

logging.json: true
logging.metrics.enabled: false

Logstash:

input {
  beats {
    port => 5046
    codec => json{charset => "ISO-8859-1"}
  }
}

filter {
  json {
    source => "message"
    target => "json"
    remove_field => ["message"]
  }
}

output {
  elasticsearch { 
    hosts => ["192.168.1.14:9200"] 
    index => "logstash-%{[@metadata][indexdate]}"
  }
  stdout { codec => rubydebug }
}

I've now reread the log for 10 times and noticed the error:
"reason"=>"Invalid index name [logstash-%{[@metadata][indexDate]}], must be lowercase", "index_uuid"=>"na", "index"=>"logstash-%{[@metadata][indexDate]}"}}}}

I corrected the index and everything worked! the index appeared in Kibane!

11

Remove this, I don't think this is needed, the beats output to logstash using utf-8 if I'm not wrong.

I don't think you need this, you are already parsing the json message in your filebeat.yml with the decode_json_fields processor.

This happens because the field [@metadata][indexDate] does not exist in some document, so it will use this literal value, which will give you the error since it has a uppercase letter.

Where is this field comming from? There is nothing in your logstash pipeline that creates it.

12

I deleted the config

{charset => "ISO-8859-1"}

I deleted this from the config and logstash didn't run because of errors, I put it back the way it was

json {
    source => "message"
    target => "json"
    remove_field => ["message"]
  }

I wasn't happy for a long time, logstash looped and started clogging the disk space fast and I don't know why, in kibane I see an error

index [.async-search] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
13

In the logs I see a parsing error and infinitely repeated values

[2023-04-19T13:54:33,530][WARN ][logstash.filters.json    ][main][e9badb283d4469c6c8ba51c6d06909a513fde1fa93cd91e27af053107ad47e60] Error parsing json {:source=>"message", :raw=>"[2023-04-19T13:54:26.943+00:00][INFO ][plugins-service] Plugin \"profiling\" is disabled.", :exception=>#<LogStash::Json::ParserError: Unexpected character ('-' (code 45)): was expecting comma to separate Array entries
 at [Source: (byte[])"[2023-04-19T13:54:26.943+00:00][INFO ][plugins-service] Plugin "profiling" is disabled."; line: 1, column: 7]>}

[2023-04-19T13:54:36,338][WARN ][logstash.filters.json    ][main][e9badb283d4469c6c8ba51c6d06909a513fde1fa93cd91e27af053107ad47e60] Error parsing json {:source=>"message", :raw=>"                               \"org_label-schema_url\" => \"https://www.elastic.co/products/beats/filebeat\"", :exception=>#<LogStash::Json::ParserError: Unexpected character ('=' (code 61)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"                               "org_label-schema_url" => "https://www.elastic.co/products/beats/filebeat""; line: 1, column: 56]>}

{
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
      "@version" => "1",
        "stream" => "stderr",
    "@timestamp" => 2023-04-19T13:54:07.595Z,
         "agent" => {
            "hostname" => "5e4a6f707fd2",
        "ephemeral_id" => "e20cbf65-fa84-497d-aeca-f159d5cd69c2",
                  "id" => "a0cda17a-2e91-4dc7-a037-a94498c490d3",
                "type" => "filebeat",
             "version" => "7.2.0"
    },
          "host" => {
        "name" => "5e4a6f707fd2"
    },
           "ecs" => {
        "version" => "1.0.0"
    },
     "container" => {
            "id" => "5e4a6f707fd2c863360c122b45e48a066ca33d9cbd744acd46c7d38db35bf56c",
          "name" => "filebeat",
        "labels" => {
                           "org_label-schema_version" => "7.2.0",
                                        "description" => "Filebeat sends log files to Logstash or directly to Elasticsearch.",
                           "org_label-schema_license" => "GPLv2",
                "com_docker_compose_container-number" => "1",
             "com_docker_compose_project_working_dir" => "/home/maks/docker/ELK/help",
                           "org_label-schema_vcs-url" => "github.com/elastic/beats",
                    "org_label-schema_schema-version" => "1.0",
                      "com_docker_compose_depends_on" => "logstash:service_started:false",
                         "com_docker_compose_service" => "filebeat",
                          "com_docker_compose_oneoff" => "False",
            "com_docker_compose_project_config_files" => "/home/maks/docker/ELK/help/docker-compose.yml",
                         "com_docker_compose_version" => "2.17.2",
                           "org_label-schema_vcs-ref" => "9ba65d864ca37cd32c25b980dbb4020975288fc0",
                                            "license" => "Elastic License",
                        "org_label-schema_build-date" => "20190305",
                              "org_label-schema_name" => "filebeat",
                            "org_label-schema_vendor" => "Elastic",
                     "com_docker_compose_config-hash" => "d4907d7bd52514db27464e7319fd7e4f0fe65d53a162b59054f7edf571395348",
                           "com_docker_compose_image" => "sha256:30ee5f4f93abd3d63181ea1706f54a1c14a61045f5a2a6a4bbf1562b13937d9a",
                               "org_label-schema_url" => "https://www.elastic.co/products/beats/filebeat",
                         "com_docker_compose_project" => "help"
        },
         "image" => {
            "name" => "elastic/filebeat:7.2.0"
        }
    },
          "json" => {
            "level" => "info",
        "timestamp" => "2023-04-19T13:54:07.595Z",
           "caller" => "instance/beat.go:292",
          "message" => "Setup Beat: filebeat; Version: 7.2.0"
    },
         "input" => {
        "type" => "container"
    },
           "log" => {
          "file" => {
            "path" => "/var/lib/docker/containers/5e4a6f707fd2c863360c122b45e48a066ca33d9cbd744acd46c7d38db35bf56c/5e4a6f707fd2c863360c122b45e48a066ca33d9cbd744acd46c7d38db35bf56c-json.log"
        },
        "offset" => 4845
    }
}
14

Do you have enough disk space? Do now let the free disk space bellow 15%. Check this

15

I set the Elasticsearch config:
cluster.routing.allocation.disk.watermark.flood_stage
But this did not help me, after deleting the container and raising a new one, the disk rapidly begins to run out of space, 30 gb in 15 minutes, and all this space is taken by the volume logstash

16

I would avoid debug mode, comment out: #stdout { codec => rubydebug }
Are you sure LS consumes a lot of space?
Are you using in-memory queue processing or persistant on disk?
Are you set log.level: info

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.