LogStash Special Character Split Error

Elastic Stack Logstash
1

Hi! I have some problem with Logstash Filter

  1. Filebeat collect -> "aaa.log" file
  2. and then message field -> "ID∮NAME∮TELNO"
  3. And Logstash.conf is

input {
beats { port => xxxx
}

filter {
mutate {
split => { "message" => "∮" }
}
}

output {
stdout {}
}

  1. But LogStash Console Output Error
    "The following config files contains non-ascii characters but are not UTF-8 encoded"

  2. Therefore.. I did some test.... Original logfile content & Logstash Conf change

  • "ID∮NAME∮TELNO" -> "ID,NAME,TELNO"
  • split => { "message" => "," }
  1. Then Logstash filter Split success

But I need "∮" special character split pattern.
Is there any way to do it?

2

Are you doing something different than this? It works for me.

Conf

input {
  generator {
    message => '[{ "message": "ID∮NAME∮TELNO" }]'
    count => 1
    codec => "json"
  }
}
filter {
  mutate {
    split => { "message" => "∮" }
  }  
}
output {
  stdout { codec =>  "json" }
}

Output

{
    "message": [
        "ID",
        "NAME",
        "TELNO"
    ],
    "host": "Aarons-MacBook-Pro.local",
    "@version": "1",
    "@timestamp": "2022-01-23T11:52:17.130Z",
    "sequence": 0
}
3

In my case Output... like this

[2022-01-23T21:09:36,704][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"The following config files contains non-ascii characters but are not UTF-8 encoded ["c:/logstash-7.16.3/config/logstash.conf"]", :backtrace=>["C:/logstash-7.16.3/logstash-core/lib/logstash/config/source/local.rb:99:in read'", "C:/logstash-7.16.3/logstash-core/lib/logstash/config/source/local.rb:110:in read'", "C:/logstash-7.16.3/logstash-core/lib/logstash/config/source/local.rb:206:in local_pipeline_configs'", "C:/logstash-7.16.3/logstash-core/lib/logstash/config/source/local.rb:177:in pipeline_configs'", "C:/logstash-7.16.3/logstash-core/lib/logstash/config/source_loader.rb:76:in block in fetch'", "org/jruby/RubyArray.java:2584:in collect'", "C:/logstash-7.16.3/logstash-core/lib/logstash/config/source_loader.rb:75:in fetch'", "C:/logstash-7.16.3/logstash-core/lib/logstash/agent.rb:182:in converge_state_and_update'", "C:/logstash-7.16.3/logstash-core/lib/logstash/agent.rb:120:in execute'", "C:/logstash-7.16.3/logstash-core/lib/logstash/runner.rb:432:in block in execute'", "C:/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

  1. Most of Special Character is Output Normal (Ex) "/", "#", "@", "!"

"message" => [
[0] "ID",
[1] "NAME",
[2] "TELNO"
]

But "§" is Error..

4

This is really weird, I simulated with the same symbol in the pipeline and it worked without any problem.

But I'm on a Linux system and from the output of @aaron-nimocks example, he is on a Mac, which is Unix based, maybe it is something related to how windows is encoding the configuration file, this can give some problems in some cases.

You could try to not use the ∮ char but use its unicode code, you would need to replace it with another character to use in the split filter.

This works:

mutate {
    gsub => ["message","\u222E","|"]
}
mutate {
    split => {"message" => "|"}
}
5

Just one thing to avoid confusion which is the character that you want to use in the split filter?

Your original message is that ID∮NAME∮TELNO, so you would split on this character ∮, which is a math symbol representing a contour integral.

But your last message you shared this other symbol §, which is a different one, this is a section sign, used to reference individual sections on a document, they look similar but are completely different.

The gsub approach still work, you just would need to use the correct code, if this is indeed the split character.

6

Which editor do you use to create logstash.conf?

"/", "#", "@", "!" are ASCII characters and could make no error.
There are some encodings which share ASCII part of character codes with UTF-8 and other part are completely different. One example is Shift-JIS in Japan and there maybe some for other CJK characters.

Please check the encoding of logstash.conf.

7

Thanks leandrojmp and Tomo_M

I followed you adivce.. my "logstash.conf" Encoding Change
and then This problem has been resolved!!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.