Logstash stopped processing because of an error: (SystemExit) exit

Elastic Stack Logstash
21

After checking the logs, it seems that no previous errors have appeared.

# tail /var/log/logstash/logstash-plain.log
[2021-06-15T02:30:30,469][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2021-06-15T02:30:30,472][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2021-06-15T02:30:30,474][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-06-15T02:30:30,494][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2021-06-15T02:30:30,687][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/logstash-its.conf"], :thread=>"#<Thread:0x6c4e39a5 run>"}
[2021-06-15T02:30:32,124][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.43}
[2021-06-15T02:30:32,149][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-06-15T02:30:32,170][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-06-15T02:30:32,350][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-06-15T02:30:32,386][INFO ][org.logstash.beats.Server][main][1ac5a4b383699ab24de3ff383cf5830d30237b5095c449ea663e12bb4f081b1c] Starting server on port: 5044
22 
# tail /var/log/messages
Jun 15 02:30:54 ITS-ELS-02 logstash: [2021-06-15T02:30:54,093][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-16", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"ef9152d9-43cc-45b4-8dc5-50d25349e2e2", "hostname"=>"MAX-CAS-16", "id"=>"de0078b3-624b-4ec4-af10-971c190d46d7"}, "tags"=>["madam", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-16", "mac"=>["00:50:56:89:3f:5e"], "architecture"=>"x86_64", "ip"=>["10.131.4.116"], "hostname"=>"MAX-CAS-16", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"room_user_hash\" : \"2be2ec464ca1553accc0c59656f6c056\"", "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>123298364}, "ecs"=>{"version"=>"1.6.0"}, "@timestamp"=>2021-06-14T17:30:37.014Z}}
Jun 15 02:30:54 ITS-ELS-02 logstash: [2021-06-15T02:30:54,096][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-16", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"ef9152d9-43cc-45b4-8dc5-50d25349e2e2", "hostname"=>"MAX-CAS-16", "id"=>"de0078b3-624b-4ec4-af10-971c190d46d7"}, "tags"=>["madam", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-16", "mac"=>["00:50:56:89:3f:5e"], "architecture"=>"x86_64", "ip"=>["10.131.4.116"], "hostname"=>"MAX-CAS-16", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"}", "ecs"=>{"version"=>"1.6.0"}, "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>123298420}, "@timestamp"=>2021-06-14T17:30:37.014Z}}
Jun 15 02:30:54 ITS-ELS-02 logstash: [2021-06-15T02:30:54,378][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "tags"=>["madam", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "agent"=>{"name"=>"MAX-CAS-16", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"ef9152d9-43cc-45b4-8dc5-50d25349e2e2", "hostname"=>"MAX-CAS-16", "id"=>"de0078b3-624b-4ec4-af10-971c190d46d7"}, "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-16", "mac"=>["00:50:56:89:3f:5e"], "architecture"=>"x86_64", "ip"=>["10.131.4.116"], "hostname"=>"MAX-CAS-16", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"type\" : \"connection_check\",", "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>123370702}, "ecs"=>{"version"=>"1.6.0"}, "@timestamp"=>2021-06-14T17:30:49.021Z}}
Jun 15 02:30:54 ITS-ELS-02 logstash: [2021-06-15T02:30:54,380][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-16", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"ef9152d9-43cc-45b4-8dc5-50d25349e2e2", "hostname"=>"MAX-CAS-16", "id"=>"de0078b3-624b-4ec4-af10-971c190d46d7"}, "tags"=>["madam", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-16", "mac"=>["00:50:56:89:3f:5e"], "architecture"=>"x86_64", "ip"=>["10.131.4.116"], "hostname"=>"MAX-CAS-16", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"room_user_hash\" : \"2be2ec464ca1553accc0c59656f6c056\",", "ecs"=>{"version"=>"1.6.0"}, "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>123370733}, "@timestamp"=>2021-06-14T17:30:49.021Z}}
Jun 15 02:30:54 ITS-ELS-02 logstash: [2021-06-15T02:30:54,381][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "tags"=>["madam", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "agent"=>{"name"=>"MAX-CAS-16", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"ef9152d9-43cc-45b4-8dc5-50d25349e2e2", "hostname"=>"MAX-CAS-16", "id"=>"de0078b3-624b-4ec4-af10-971c190d46d7"}, "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-16", "mac"=>["00:50:56:89:3f:5e"], "architecture"=>"x86_64", "ip"=>["10.131.4.116"], "hostname"=>"MAX-CAS-16", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"room_hash\" : \"097bd5a604efcf81f61c4bdc1343ca79\"", "ecs"=>{"version"=>"1.6.0"}, "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>123370790}, "@timestamp"=>2021-06-14T17:30:49.021Z}}
Jun 15 02:30:54 ITS-ELS-02 logstash: [2021-06-15T02:30:54,381][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-16", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"ef9152d9-43cc-45b4-8dc5-50d25349e2e2", "hostname"=>"MAX-CAS-16", "id"=>"de0078b3-624b-4ec4-af10-971c190d46d7"}, "tags"=>["madam", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-16", "mac"=>["00:50:56:89:3f:5e"], "architecture"=>"x86_64", "ip"=>["10.131.4.116"], "hostname"=>"MAX-CAS-16", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"}", "ecs"=>{"version"=>"1.6.0"}, "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>123370841}, "@timestamp"=>2021-06-14T17:30:49.021Z}}
Jun 15 02:30:55 ITS-ELS-02 logstash: [2021-06-15T02:30:55,940][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-03", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"b12f3643-4df1-4efb-8fd6-5b8981f10c54", "hostname"=>"MAX-CAS-03", "id"=>"6f6249ec-837d-469a-9663-443dbcdc29fe"}, "tags"=>["jewel", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-03", "mac"=>["00:50:56:89:fd:12"], "architecture"=>"x86_64", "ip"=>["10.132.4.140"], "hostname"=>"MAX-CAS-03", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"type\" : \"connection_check\",", "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>132203696}, "ecs"=>{"version"=>"1.6.0"}, "@timestamp"=>2021-06-14T17:30:54.929Z}}
Jun 15 02:30:55 ITS-ELS-02 logstash: [2021-06-15T02:30:55,941][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-03", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"b12f3643-4df1-4efb-8fd6-5b8981f10c54", "hostname"=>"MAX-CAS-03", "id"=>"6f6249ec-837d-469a-9663-443dbcdc29fe"}, "tags"=>["jewel", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-03", "mac"=>["00:50:56:89:fd:12"], "architecture"=>"x86_64", "ip"=>["10.132.4.140"], "hostname"=>"MAX-CAS-03", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"room_hash\" : \"8b4588fb28485cdadfa4094c54eaede0\",", "ecs"=>{"version"=>"1.6.0"}, "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>132203727}, "@timestamp"=>2021-06-14T17:30:54.929Z}}
Jun 15 02:30:55 ITS-ELS-02 logstash: [2021-06-15T02:30:55,942][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "agent"=>{"name"=>"MAX-CAS-03", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"b12f3643-4df1-4efb-8fd6-5b8981f10c54", "hostname"=>"MAX-CAS-03", "id"=>"6f6249ec-837d-469a-9663-443dbcdc29fe"}, "tags"=>["jewel", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-03", "mac"=>["00:50:56:89:fd:12"], "architecture"=>"x86_64", "ip"=>["10.132.4.140"], "hostname"=>"MAX-CAS-03", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"  \"room_user_hash\" : \"68a1740df20e3b898e02969c8b314343\"", "ecs"=>{"version"=>"1.6.0"}, "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>132203779}, "@timestamp"=>2021-06-14T17:30:54.929Z}}
Jun 15 02:30:55 ITS-ELS-02 logstash: [2021-06-15T02:30:55,942][WARN ][org.logstash.dissect.Dissector][main][c808e03b37c187ee2fd502a1ed2197bf694f0861808ee9ef4946c07b4096d95e] Dissector mapping, pattern not found {"field"=>"message", "pattern"=>"%{timestamp}|[%{thread-name}]|%{log-level}|%{class-name} - %{log-message}", "event"=>{"@version"=>"1", "tags"=>["jewel", "cas", "server", "beats_input_codec_plain_applied", "_dissectfailure"], "agent"=>{"name"=>"MAX-CAS-03", "version"=>"7.11.1", "type"=>"filebeat", "ephemeral_id"=>"b12f3643-4df1-4efb-8fd6-5b8981f10c54", "hostname"=>"MAX-CAS-03", "id"=>"6f6249ec-837d-469a-9663-443dbcdc29fe"}, "input"=>{"type"=>"log"}, "host"=>{"containerized"=>false, "name"=>"MAX-CAS-03", "mac"=>["00:50:56:89:fd:12"], "architecture"=>"x86_64", "ip"=>["10.132.4.140"], "hostname"=>"MAX-CAS-03", "id"=>"05bb365618374cc39bcb5d149f7229b9", "os"=>{"kernel"=>"3.10.0-1062.18.1.el7.x86_64", "codename"=>"Core", "name"=>"CentOS Linux", "version"=>"7 (Core)", "platform"=>"centos", "family"=>"redhat"}}, "message"=>"}", "log"=>{"file"=>{"path"=>"/var/log/cas/server.log"}, "offset"=>132203835}, "ecs"=>{"version"=>"1.6.0"}, "@timestamp"=>2021-06-14T17:30:54.929Z}}

It seems that no Index with today's date has been created yet, so we will have to wait and see.

23

I have since confirmed that the logs are being collected smoothly.
Thank you very much.
The way you taught me is more correct, since I am using the rpm package for the boot method.

One last question.
I used to use bin/logstash because --config.reload.automatic was very useful.

In the case of systemctl starat logstash, just editing the conf file didn't work, so I had to turn off the process again and start it, is there a better way?
By the way, systemctl restart logstash didn't work either.

24

If you have a pipeline error you'll sometimes have to kill the process by hand in order to unlock temporary queing files that are accumulating because of the failing pipeline when using systemctl restart.

This will result in a loss of logs depending on the configuration.

25

I see.
So I had to kill the process.

I want to avoid losing logs as much as possible.
Is there a better way?

26

You can check logstash path.data configuration and this article

1 Like
27

Thank you very much. That is a good idea.

I'll set queue.type: persisted and see how it goes for a while.

28

It has been running stably for about a day since I changed the settings.
I have not yet tested whether data is lost when logstash is stopped, but I will ask the question in another case if there is a problem.

Thank you all for your help.

29

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.