Erreur d'importation avec logstash sous linux

J'importe mes logs avec logstash vers elasticserach, ça marche bien en local ( sous windows).
Mais pas sous linux, j'ai pleins d'erreurs et je ne sais pas pourquoi!
j'ai remarqué que le dans fichier.txt , s'est écrit "convertie en DOS" je me dis donc que peut être le format a changé.
J'ai remarqué aussi qu'un ligne s'importe avec succès et la suivante non, et il y a des \\t qui apparaissent à la place de \t.
Est ce que vous pourriez m'aider ?

Exemple du log.txt

20191101  00:00:00.390  MPC CarteAbsente A:3,I:1,P:0,R:0,E:1
20191101  00:00:00.500  MPC GereRecepMPC10 | EOT1 - Msg Reçu :Aa0040000a A:6,I:1,P:0,R:1,E:1

Exemple du log.conf

grok {
	match => ["message","(?<timestamp>%{YEAR:YYYY}%{MONTHNUM:MM}%{MONTHDAY:dd}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}.[0-9]{3})%{SPACE}%{GREEDYDATA:data}"]
	date {
		match => ["timestamp", "YYYYMMdd        HH:mm:ss.SSS", "YYYYMMdd HH:mm:ss.SSS"]
		target => "@timestamp"
		timezone => "GMT"
		locale => "fr"

If you want to ask a question in French then you may get a better answer in another forum. I did a Google translate and did not really understand the question about tab conversion. A native speaker may well understand better.

Can I delete this post and ask another the same question in english ?

I import my logs with logstash to elasticserach, it works well locally (under windows).
But not under Linux, I have lots of errors and I don't know why!
I noticed that the in file.txt, was written "converted to DOS" so I tell myself that maybe the format has changed.
I also noticed that one line imports successfully and the next does not, and there are \ t that appear in place of \ t.
Can you help me?
Thank you.

Can you post the errors you are getting? If you output the log file in a terminal in linux (cat file.txt) does it correctly print multiple lines?

There is a difference in the message field

          "data" => "MPC CarteAbsente A:3,I:1,P:0,R:0,E:1",
     "timestamp" => "20191101\t04:24:38.484",
       "message" => "20191101\t04:24:38.484\tMPC CarteAbsente A:3,I:1,P:0,R:0,E:1",
    "@timestamp" => 2019-11-01T04:24:38.484Z,
           "seq" => 58175,
          "host" => "ns510529"
       "message" => "20191101\\t04:24:38.593\\tMPC GereRecepMPC10 | EOT1 - Msg Re\\xE7u :\\u0002Aa0040000\\u0010\\u0003\\a A:6,I:1,P:0,R:1,E:1",
    "@timestamp" => 2020-05-20T08:18:53.952Z,
          "tags" => [
        [0] "_grokparsefailure"
           "seq" => 58176,
          "host" => "ns510529"

I am not 100% sure but i think the issue is with parsing the unicode characters in that second line

I think the \\t in the message is misleading as on failure grok will put the entire line in the message and is probably re escaping the \t in the log line

I think you need to specify the charset as UTF-8 - but Im not sure how to do that. Something like

1 Like

I found this :

codec => plain {
      charset => "name of charset here"

but having already used

codec => multiline {

I don't know how to use both?

You cannot use both, pick one or the other.

1 Like

I tried

codec => multiline {
                 charset  => "UTF-8",


But in this case, logstash don't run !!!

Remove the trailing comma.

1 Like

i still have error message and logstash is wont run!

What error message?

I can't understand it!

warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/
             add at org/logstash/execution/
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:355
[ERROR] 2020-05-27 22:24:51.414 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ `create'", "org/logstash/execution/ `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in `block in converge_state'"]}
[FATAL] 2020-05-27 22:24:51.437 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ `create'", "org/logstash/execution/ `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in `block in converge_state'"]}
[ERROR] 2020-05-27 22:24:51.452 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

and my code is

        file {
        path => ["/home/katia/FichiersLog/SAMIA/h8.txt"]
                start_position => beginning
                codec => multiline {
                        charset => 'ASCII_8BIT'
                        pattern => "^%{YEAR}%{MONTHNUM2}%{MONTHDAY}"
                        negate => true
                        what => previous

Can you try adding

--java-execution false

to the command line and see if you get a better error message?

1 Like
[INFO ] 2020-05-28 01:24:11.830 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.0"}
[ERROR] 2020-05-28 01:24:13.364 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 6, column 12 (byte 119) after input{\n\tfile {\n        path => [\"/home/katia/g2.txt\"]\n\t\tstart_position => beginning\n\t\tcodec => multiline {\n\t\t\tcharset ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/ `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/ `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:40:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}
[INFO ] 2020-05-28 01:24:13.678 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-28 01:24:18.555 [LogStash::Runner] runner - Logstash shut down.

Now it's officially an encryption problem.
Problem lines contain characters like: ç ,ô, ' ...
and, in my CentOS, those characters appear like this "?"

What exactly comes after charset in your multiline codec configuration? An error could be caused by it being = rather than =>

1 Like

Ops, that's right. I put = and not =>
Now logstash is running, i still have some warning like :

[WARN ] 2020-05-28 02:05:43.777 [[main]<file] multiline - Received an event that has a different character encoding than you configured. {:text=>"20191101\\t02:42:10.421\\tMPC GereRecepMPC10 | EOT1 - Msg Re\\xE7u :\\u0002Aa0040000\\u0010\\u0003\\a A:6,I:1,P:0,R:1,E:1\\r", :expected_charset=>"UTF-8"}

We can see that "Msg Reçu" becomes Msg Re\xE7u...
I think I need to put something other than UTF-8

I agree. I am not an expert in character encodings so I cannot tell you what else to use.

Problem solved.
my final code is:

        file {
        path => ["/home/katia/FichiersLog/SAMIA/gest1.txt"]
                start_position => beginning
                codec => multiline {
                        charset => "BINARY"
                        pattern => "^%{YEAR}%{MONTHNUM2}%{MONTHDAY}"
                        negate => true
                        what => previous

thank you very much for your time.