Erreur d'importation avec logstash sous linux

Katia · May 20, 2020, 12:28am

Bonjour,
J'importe mes logs avec logstash vers elasticserach, ça marche bien en local ( sous windows).
Mais pas sous linux, j'ai pleins d'erreurs et je ne sais pas pourquoi!
j'ai remarqué que le dans fichier.txt , s'est écrit "convertie en DOS" je me dis donc que peut être le format a changé.
J'ai remarqué aussi qu'un ligne s'importe avec succès et la suivante non, et il y a des \\t qui apparaissent à la place de \t.
Est ce que vous pourriez m'aider ?
Merci.

Exemple du log.txt

20191101  00:00:00.390  MPC CarteAbsente A:3,I:1,P:0,R:0,E:1
20191101  00:00:00.500  MPC GereRecepMPC10 | EOT1 - Msg Reçu :Aa0040000a A:6,I:1,P:0,R:1,E:1

Exemple du log.conf

grok {
	match => ["message","(?<timestamp>%{YEAR:YYYY}%{MONTHNUM:MM}%{MONTHDAY:dd}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}.[0-9]{3})%{SPACE}%{GREEDYDATA:data}"]
	}
	date {
		match => ["timestamp", "YYYYMMdd        HH:mm:ss.SSS", "YYYYMMdd HH:mm:ss.SSS"]
		target => "@timestamp"
		timezone => "GMT"
		locale => "fr"
	}

Badger · May 20, 2020, 1:39am

If you want to ask a question in French then you may get a better answer in another forum. I did a Google translate and did not really understand the question about tab conversion. A native speaker may well understand better.

Katia · May 20, 2020, 7:24am

Can I delete this post and ask another the same question in english ?

mcardlesam · May 20, 2020, 7:32am

Hello,
I import my logs with logstash to elasticserach, it works well locally (under windows).
But not under Linux, I have lots of errors and I don't know why!
I noticed that the in file.txt, was written "converted to DOS" so I tell myself that maybe the format has changed.
I also noticed that one line imports successfully and the next does not, and there are \ t that appear in place of \ t.
Can you help me?
Thank you.

Can you post the errors you are getting? If you output the log file in a terminal in linux (cat file.txt) does it correctly print multiple lines?

Katia · May 20, 2020, 8:22am

There is a difference in the message field

{
          "data" => "MPC CarteAbsente A:3,I:1,P:0,R:0,E:1",
     "timestamp" => "20191101\t04:24:38.484",
       "message" => "20191101\t04:24:38.484\tMPC CarteAbsente A:3,I:1,P:0,R:0,E:1",
    "@timestamp" => 2019-11-01T04:24:38.484Z,
           "seq" => 58175,
          "host" => "ns510529"
}
{
       "message" => "20191101\\t04:24:38.593\\tMPC GereRecepMPC10 | EOT1 - Msg Re\\xE7u :\\u0002Aa0040000\\u0010\\u0003\\a A:6,I:1,P:0,R:1,E:1",
    "@timestamp" => 2020-05-20T08:18:53.952Z,
          "tags" => [
        [0] "_grokparsefailure"
    ],
           "seq" => 58176,
          "host" => "ns510529"
}

mcardlesam · May 20, 2020, 9:36am

I am not 100% sure but i think the issue is with parsing the unicode characters in that second line

I think the \\t in the message is misleading as on failure grok will put the entire line in the message and is probably re escaping the \t in the log line

I think you need to specify the charset as UTF-8 - but Im not sure how to do that. Something like
https://www.elastic.co/guide/en/logstash/current/plugins-codecs-plain.html

Katia · May 21, 2020, 10:01am

I found this :

codec => plain {
      charset => "name of charset here"
    }

but having already used

codec => multiline {
		}

I don't know how to use both?

Badger · May 21, 2020, 2:52pm

You cannot use both, pick one or the other.

Katia · May 21, 2020, 3:16pm

I tried

codec => multiline {
                 charset  => "UTF-8",

		}

But in this case, logstash don't run !!!

Badger · May 21, 2020, 3:20pm

Remove the trailing comma.

Katia · May 27, 2020, 7:53pm

i still have error message and logstash is wont run!

Badger · May 27, 2020, 8:11pm

What error message?

Katia · May 27, 2020, 8:28pm

I can't understand it!

warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:129
             add at org/logstash/execution/ConvergeResultExt.java:57
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:355
[ERROR] 2020-05-27 22:24:51.414 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in `block in converge_state'"]}
[FATAL] 2020-05-27 22:24:51.437 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:355:in `block in converge_state'"]}
[ERROR] 2020-05-27 22:24:51.452 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

and my code is

input{
        file {
        path => ["/home/katia/FichiersLog/SAMIA/h8.txt"]
                start_position => beginning
                codec => multiline {
                        charset => 'ASCII_8BIT'
                        pattern => "^%{YEAR}%{MONTHNUM2}%{MONTHDAY}"
                        negate => true
                        what => previous
                }
    }
}

Badger · May 27, 2020, 9:04pm

Can you try adding

--java-execution false

to the command line and see if you get a better error message?

Katia · May 27, 2020, 11:27pm

[INFO ] 2020-05-28 01:24:11.830 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.0"}
[ERROR] 2020-05-28 01:24:13.364 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 6, column 12 (byte 119) after input{\n\tfile {\n        path => [\"/home/katia/g2.txt\"]\n\t\tstart_position => beginning\n\t\tcodec => multiline {\n\t\t\tcharset ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:40:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}
[INFO ] 2020-05-28 01:24:13.678 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-28 01:24:18.555 [LogStash::Runner] runner - Logstash shut down.

Katia · May 27, 2020, 11:39pm

Now it's officially an encryption problem.
Problem lines contain characters like: ç ,ô, ' ...
and, in my CentOS, those characters appear like this "?"

Badger · May 27, 2020, 11:48pm

What exactly comes after charset in your multiline codec configuration? An error could be caused by it being = rather than =>

Katia · May 28, 2020, 12:12am

Ops, that's right. I put = and not =>
Now logstash is running, i still have some warning like :

[WARN ] 2020-05-28 02:05:43.777 [[main]<file] multiline - Received an event that has a different character encoding than you configured. {:text=>"20191101\\t02:42:10.421\\tMPC GereRecepMPC10 | EOT1 - Msg Re\\xE7u :\\u0002Aa0040000\\u0010\\u0003\\a A:6,I:1,P:0,R:1,E:1\\r", :expected_charset=>"UTF-8"}

We can see that "Msg Reçu" becomes Msg Re\xE7u...
I think I need to put something other than UTF-8

Badger · May 28, 2020, 12:32am

I agree. I am not an expert in character encodings so I cannot tell you what else to use.

Katia · May 28, 2020, 12:37am

Problem solved.
my final code is:

input{
        file {
        path => ["/home/katia/FichiersLog/SAMIA/gest1.txt"]
                start_position => beginning
                codec => multiline {
                        charset => "BINARY"
                        pattern => "^%{YEAR}%{MONTHNUM2}%{MONTHDAY}"
                        negate => true
                        what => previous
                }
    }
}

thank you very much for your time.