# Logstash shut down and stopped processing because of an error: (SystemExit)

**URL:** <https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [February 28, 2024, 12:11am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280 "2024-02-28T00:11:16Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mad\_User](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_user/32/132136_2.png) [@Mad\_User](https://discuss.elastic.co/u/Mad_User)\
**Post date:** [February 28, 2024, 12:11am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/1 "2024-02-28T00:11:16Z")

</div>

Let me say right away that I am not a professional. I have a problem with Logstash. A few days ago it stopped collecting data and I found an error in the logs. Before it stopped collecting data, changes were made to logstash.conf, but I had a backup copy of logstash.conf on which everything initially worked. Therefore, I checked with a backup copy of logstash.conf - but the error still remained. Please tell me what else I can check?

```auto
[docker-elk-logstash-1 |[0m [2024-02-27T20:17:38,066][INFO][logstash.runner] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
    [docker-elk-logstash-1 |[0m [2024-02-27T20:17:38,161][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.10.2", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.8+7 on 17.0.8+7 +indy +jit [x86_64-linux]"}
    [docker-elk-logstash-1 |[0m [2024-02-27T20:17:38,164][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, 
        -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, - Djava.security.egd=file:/dev/urandom, 
        -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, - Dls.cgroup.cpu.path.override=/, 
        -Xms256m, -Xmx256m, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, 
        --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add- exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, 
        --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add- exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, 
        --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add- opens=java.base/java.security=ALL-UNNAMED, 
        --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL- UNNAMED, 
        --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add- opens=java.management/sun.management=ALL-UNNAMED]
    [docker-elk-logstash-1 |[0m [2024-02-27T20:17:54,528][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
    [docker-elk-logstash-1 |[0m [2024-02-27T20:19:15,265][ERROR][logstash.agent] Failed to execute action {
    :action=>LogStash::PipelineAction::Create/pipeline_id:main, 
    :exception=>"LogStash::ConfigurationError", 
    :message=>"Expected one of [\\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", 
    :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", 
             "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", 
             "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", 
             "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", 
             "org/jruby/RubyClass.java:931:in `new'", 
             "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", 
             "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
    [docker-elk-logstash-1 |[0m [2024-02-27T20:19:15,569][INFO][logstash.runner] Logstash shut down.
    [docker-elk-logstash-1 |[0m [2024-02-27T20:19:15,616][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
    [docker-elk-logstash-1 |[0m org.jruby.exceptions.SystemExit: (SystemExit) exit
    [docker-elk-logstash-1 |[0m at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~ [jruby.jar:?]
    [docker-elk-logstash-1 |[0m at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~ [jruby.jar:?]
    [docker-elk-logstash-1 |[0m at usr.share.logstash.lib.bootstrap.environment.<main> (/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
    [docker-elk-logstash-1 |[0m Using bundled JDK: /usr/share/logstash/jdk
    [docker-elk-logstash-1 |[0m Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties

```

Our logstash file configuration looks like this:

```auto
input {
      beats {
        port => 5044
      }
    }
        filter {
      if [message] =~ /Error/ {
        grok {
          match => { "message" => ["(?:Error:(?<error_exception>.*))"] }
        }
        mutate {
          add_field => { "[@metadata][zabbix_host_error]" => "%{[fields][hostname]}" }
          add_field => { "[@metadata][zabbix_key_error]" => "gate_error" }
          add_field => { "[@metadata][zabbix_msg_error]" => "%{message}" }
        }
      }
      }
    output {
      stdout { codec => rubydebug }
      } else if [message] =~ /Error/ {
        elasticsearch {
          hosts => ["elasticsearch:9200"]
          user => "logstash_internal"
          password => "${LOGSTASH_INTERNAL_PASSWORD}"
          index => "error-%{+yyyy.MM.dd}"
        }
        zabbix {
          zabbix_host => "[@metadata][zabbix_host_error]"
          zabbix_server_host => "my_IP"
          zabbix_server_port => my_port
          zabbix_key => "[@metadata][zabbix_key_error]"
          zabbix_value => "[@metadata][zabbix_msg_error]"
        }
      } 

```

I understand that the log says that the error is in:

`message=>"Expected one of [\\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)"`

But I don't see any error there..(

Tried:

- launching with a known working config;
- docker restart container;
- completely restart the cluster;
- docker compose down && docker compose up --build -d.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 28, 2024, 1:12am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/2 "2024-02-28T01:12:38Z")

</div>

What does your docker-compose looks like?

This error normally means that something is wrong in the logstash configuration file, but since it is pointing to line 1, column 1 it probably means that logstash is trying to load a file that is not a logstash configuration pipeline, or there is something wrong in the first line.

Also, it is probably not the issue, but the configuration you shared is wrong, you have an `else if` inside of the `output` block without an `if` first, so this is not a working config.

---

<div class="post-metadata">

**Author:** ![Mad\_User](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_user/32/132136_2.png) [@Mad\_User](https://discuss.elastic.co/u/Mad_User)\
**Post date:** [February 28, 2024, 1:24am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/3 "2024-02-28T01:24:38Z")

</div>

I'm sorry, but of course it's like that:  
`} if [message] =~ /Error/ {`  
I just corrected the file a little and this error is not in production.

Our docker-compose looks like this:

```auto
version: '3.7'

services:
  setup:
    profiles:
      - setup
    build:
      context: setup/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    init: true
    volumes:
      - ./setup/entrypoint.sh:/entrypoint.sh:ro,Z
      - ./setup/lib.sh:/lib.sh:ro,Z
      - ./setup/roles:/roles:ro,Z
    environment:
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}
      METRICBEAT_INTERNAL_PASSWORD: ${METRICBEAT_INTERNAL_PASSWORD:-}
      FILEBEAT_INTERNAL_PASSWORD: ${FILEBEAT_INTERNAL_PASSWORD:-}
      HEARTBEAT_INTERNAL_PASSWORD: ${HEARTBEAT_INTERNAL_PASSWORD:-}
      MONITORING_INTERNAL_PASSWORD: ${MONITORING_INTERNAL_PASSWORD:-}
      BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}
    networks:
      - docker-elk_elk
    depends_on:
      - elasticsearch

  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    # deploy:
    # resources:
    # limits:
    # cpus: '0.50'
    # memory: 50M
    # reservations:
    # cpus: '0.25'
    # memory: 20M
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,Z
      - elasticsearch:/usr/share/elasticsearch/data:Z
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      node.name: elasticsearch
      ES_JAVA_OPTS: -Xms1024m -Xmx1024m
      # Bootstrap password.
      # Used to initialize the keystore during the initial startup of
      # Elasticsearch. Ignored on subsequent runs.
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
    networks:
      - docker-elk_elk
    restart: unless-stopped

  logstash:
    build:
      context: logstash/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
      - ./logstash/certs:/usr/share/logstash/certs:ro,Z
    ports:
      - 5044:5044
      - 50000:50000/tcp
      - 50000:50000/udp
      - 9600:9600
    environment:
      LS_JAVA_OPTS: -Xms256m -Xmx256m
      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
    networks:
      - docker-elk_elk
    depends_on:
      - elasticsearch
    restart: unless-stopped

  kibana:
    build:
      context: kibana/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
    ports:
      - 5601:5601
    environment:
      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}
    networks:
      - docker-elk_elk
    depends_on:
      - elasticsearch
    restart: unless-stopped

  nginx:
    container_name: 'nginx-reverse-proxy'
    build:
      context: .
      dockerfile: docker/nginx.Dockerfile
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./config:/config
      - /etc/letsencrypt:/etc/letsencrypt:ro
      - /tmp/acme_challenge:/tmp/acme_challenge
    restart: always
    networks:
      - docker-elk_elk
    depends_on:
      - elasticsearch

networks:
  docker-elk_elk:
    driver: bridge

volumes:
  elasticsearch:

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 28, 2024, 1:33am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/4 "2024-02-28T01:33:42Z")

</div>

> [@Mad\_User](#):
>
> `./logstash/pipeline`

What files do you have in this path in your docker host? It can only have logstash configurations.

Please run a `ls` on this path and share it.

---

<div class="post-metadata">

**Author:** ![Mad\_User](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_user/32/132136_2.png) [@Mad\_User](https://discuss.elastic.co/u/Mad_User)\
**Post date:** [February 28, 2024, 1:47am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/5 "2024-02-28T01:47:01Z")

</div>

![Screen 2024-02-28](https://us1.discourse-cdn.com/elastic/original/3X/2/f/2f1eee32e7ce55c76b6aa997c1c1ffb382353f61.jpeg)

```auto
~/docker-elk/logstash$ ls -l
total 16
-rw-rw-r-- 1 ubuntu ubuntu 264 Oct 4 10:03 Dockerfile
drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 24 10:16 certs
drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 24 08:02 config
drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 16 14:57 pipeline

```

```auto
~/docker-elk/logstash/pipeline$ ls -l
total 3892
-rw-rw-r-- 1 ubuntu ubuntu 10375 Feb 27 21:03 logstash.conf
-rw-r--r-- 1 ubuntu ubuntu 3973120 Feb 16 14:59 stopout

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 28, 2024, 1:53am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/6 "2024-02-28T01:53:36Z")

</div>

> [@Mad\_User](#):
>
> `-rw-r--r-- 1 ubuntu ubuntu 3973120 Feb 16 14:59 stopout`

What is this file? And what is the content of it?

If it is not a logstash configuration you need to remove it from the path.

---

<div class="post-metadata">

**Author:** ![Mad\_User](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_user/32/132136_2.png) [@Mad\_User](https://discuss.elastic.co/u/Mad_User)\
**Post date:** [February 28, 2024, 2:13am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/7 "2024-02-28T02:13:43Z")

</div>

It contained (I don’t know why 🤔) some filebeat logs.  
I don't understand where it came from, but I deleted it.

---

<div class="post-metadata">

**Author:** ![Mad\_User](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mad_user/32/132136_2.png) [@Mad\_User](https://discuss.elastic.co/u/Mad_User)\
**Post date:** [February 28, 2024, 2:28am UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/8 "2024-02-28T02:28:24Z")

</div>

After deleting that file, I restarted the container and the “Logstash shut down” error disappeared from the Logstash logs.  
And what’s more, data has begun to be collected!  
You are a true professional and my savior! 🦾 🏆  
Thank you very much and have a nice day! 🌞

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 29, 2024, 9:57pm UTC](https://discuss.elastic.co/t/logstash-shut-down-and-stopped-processing-because-of-an-error-systemexit/354280/10 "2024-03-29T21:57:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.