Why do errors occur when running in a kubernetes cluster?

Elastic Stack Logstash
1

After I run Logstash in a Kubernetes cluster with the settings of
input {

   gelf {
          port => 7003
          type => "some-type-1"
      }
   gelf {
          port => 7004
          type => "some-type-2"
      }

   gelf {
          port => 7014
          type => "some-type-3"
          use_tcp => true
      }
}

I see errors in the logs when running Pod.

[ERROR] 2025-04-03 11:14:49.447 [Ruby-0-Thread-132: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 14)): only regular white space (\r, \n, \t) is allowed between tokens
 at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x0E\""}

Could it be that Kubernetes checks the port and logstash detects this as incoming logs?
If I delete the Service manifest, the errors disappear.

If I create service manifest again I see errors:

`[ERROR] 2025-04-03 15:27:09.201 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Unexpected character ((CTRL-CHAR, code 141)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')`

` at [Source: (byte[])"�"; line: 1, column: 2]>, :data=>"\"\\x8D\""}`

`[ERROR] 2025-04-03 15:27:09.201 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 5)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x05\""}`

`[ERROR] 2025-04-03 15:27:09.201 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 5)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x05\\x01\""}`

`[ERROR] 2025-04-03 15:27:09.202 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 8)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])"; line: 1, column: 2]>, :data=>"\"\\b\""}`

`[ERROR] 2025-04-03 15:27:09.202 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 29)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x1D\""}`

`[ERROR] 2025-04-03 15:27:09.202 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 23)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x17\""}`

`[ERROR] 2025-04-03 15:27:09.202 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 24)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x18\""}`

`[ERROR] 2025-04-03 15:27:09.202 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 25)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x19\""}`

`[ERROR] 2025-04-03 15:27:09.202 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 11)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\v\""}`

`[ERROR] 2025-04-03 15:27:09.203 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 2)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x02\\x01\""}`

`[ERROR] 2025-04-03 15:27:09.203 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 26)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x1A\""}`

`[ERROR] 2025-04-03 15:27:09.203 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 24)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])"�"; line: 1, column: 2]>, :data=>"\"\\x18\\b\\x04\\x04\\x03\\b\\a\\b\\x05\\b\\x06\\x04\\x01\\x05\\x01\\x06\\x01\\x05\\x03\\x06\\x03\\x02\\x01\\x02\\x03\\xFF\\x01\""}`

`[ERROR] 2025-04-03 15:27:09.203 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 1)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x01\""}`

`[ERROR] 2025-04-03 15:27:09.203 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 23)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x17\""}`

`[ERROR] 2025-04-03 15:27:09.204 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 16)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x10\""}`

`[ERROR] 2025-04-03 15:27:09.204 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 14)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x0E\""}`

`[ERROR] 2025-04-03 15:27:09.204 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 12)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])"hhttp/1.1"; line: 1, column: 2]>, :data=>"\"\\f\\x02h2\\bhttp/1.1\""}`

`[ERROR] 2025-04-03 15:27:09.204 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 18)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x12\""}`

`[ERROR] 2025-04-03 15:27:09.204 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input in a Number value`

` at [Source: (byte[])"+"; line: 1, column: 3]>, :data=>"\"+\""}`

`[ERROR] 2025-04-03 15:27:09.204 [Ruby-0-Thread-209: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 5)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x05\\x04\\x03\\x04\\x03\\x03\""}`

`[ERROR] 2025-04-03 15:27:09.518 [Ruby-0-Thread-210: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 22)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x16\\x03\\x01\\x01\""}`

`[ERROR] 2025-04-03 15:27:09.519 [Ruby-0-Thread-210: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 1)): only regular white space (\r, \n, \t) is allowed between tokens`

` at [Source: (byte[])""; line: 1, column: 2]>, :data=>"\"\\x01\""}`

`[ERROR] 2025-04-03 15:27:09.519 [Ruby-0-Thread-210: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Unrecognized token 'üH': was expecting ('true', 'false' or 'null')`

` at [Source: (byte[])"�H.���@�x�᤟JC�3�*W><���e� 3�q">Z�o@5@`���&̰�\Il(r�(���e"; line: 1, column: 6]>, :data=>"\"\\xFC\\x03\\x03H.\\x9E\\xD9\\xC6@\\xA1\\x01x\\xB6\\xE1\\xA4\\x9F\\x1AJC\\x12\\x9F3\\x06\\xE7*W\\x11><\\xB9\\xEE\\x01\\x8Be\\xCE 3\\xDFq\\\">Z\\xA8o@5@`\\x9A\\xBE\\x97&\\xCC\\xB0\\xF4\\\\I\\x02l(r\\x16\\xE4(\\x8E\\x8A\\xB5\\e\""}`

`[ERROR] 2025-04-03 15:27:09.520 [Ruby-0-Thread-210: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:127] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserErr`
2

Please change to: type => "some-type-3"

1 Like