Logstash shut down after pipeline terminated

Elastic Stack Logstash
1

i,m running elastic stack on docker and with ssl security enable.
this is my logstash.yml file

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "https://elasticsearch:9200" ]
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: *****
path.config: /etc/logstash/conf.d/*.conf
xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/elasticsearch/config/certificates/ca/ca.crt

and pipeline.conf

input {
	tcp {
		port => 5000
	}
}

output {
	elasticsearch {
		hosts => "https://elasticsearch:9200"
		user => "logstash_writer"
		cacert => '/usr/share/elasticsearch/config/certificates/ca/ca.crt'
		password => ******
	}
}

i created logstash_writer user with required roles . when i start logstash container, i face to this message

2020-11-16T08:34:31.120315588Z Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2020-11-16T08:34:31.256748933Z [2020-11-16T08:34:31,254][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +indy +jit [linux-x86_64]"}
2020-11-16T08:34:31.575792235Z [2020-11-16T08:34:31,575][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2020-11-16T08:34:32.148044896Z [2020-11-16T08:34:32,147][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
2020-11-16T08:34:32.148068535Z Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
2020-11-16T08:34:32.148073283Z https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
2020-11-16T08:34:32.969676383Z [2020-11-16T08:34:32,964][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@elasticsearch:9200/]}}
{:url=>"https://logstash_system:xxxxxx@elasticsearch:9200/"}
2020-11-16T08:34:33.586265124Z [2020-11-16T08:34:33,585][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
2020-11-16T08:34:33.589928143Z [2020-11-16T08:34:33,589][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2020-11-16T08:34:33.724245573Z [2020-11-16T08:34:33,724][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
2020-11-16T08:34:33.725057874Z [2020-11-16T08:34:33,724][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
2020-11-16T08:34:33.815155638Z [2020-11-16T08:34:33,814][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
2020-11-16T08:34:34.930455484Z [2020-11-16T08:34:34,930][INFO ][org.reflections.Reflections] Reflections took 40 ms to scan 1 urls, producing 22 keys and 45 values 
2020-11-16T08:34:35.091871327Z [2020-11-16T08:34:35,091][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
2020-11-16T08:34:35.091908736Z ** WARNING ** You have enabled encryption but DISABLED certificate verification.
2020-11-16T08:34:35.091917237Z ** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
2020-11-16T08:34:35.148184203Z [2020-11-16T08:34:35,147][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@elasticsearch:9200/]}}
2020-11-16T08:34:35.200325984Z [2020-11-16T08:34:35,200][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@elasticsearch:9200/"}
2020-11-16T08:34:35.209871289Z [2020-11-16T08:34:35,209][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es_version=>7}
2020-11-16T08:34:35.210524630Z [2020-11-16T08:34:35,210][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2020-11-16T08:34:35.255195494Z [2020-11-16T08:34:35,254][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["https://elasticsearch:9200"]}
2020-11-16T08:34:35.276955888Z [2020-11-16T08:34:35,276][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
2020-11-16T08:34:35.346930351Z [2020-11-16T08:34:35,346][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x19f9b5d0 run>"}
2020-11-16T08:34:35.959862754Z [2020-11-16T08:34:35,959][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.61}
2020-11-16T08:34:35.984941406Z [2020-11-16T08:34:35,984][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
2020-11-16T08:34:36.042730614Z [2020-11-16T08:34:36,042][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
2020-11-16T08:34:36.208647874Z [2020-11-16T08:34:36,208][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
2020-11-16T08:34:38.195292749Z [2020-11-16T08:34:38,194][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
2020-11-16T08:34:38.229489078Z [2020-11-16T08:34:38,229][INFO ][logstash.runner          ] Logstash shut down.
2020-11-16T08:34:39.085339262Z OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020-11-16T08:34:53.831760691Z WARNING: An illegal reflective access operation has occurred
2020-11-16T08:34:53.831803182Z WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby15212740039221620722jopenssl.jar) to field java.security.MessageDigest.provider
2020-11-16T08:34:53.831809551Z WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
2020-11-16T08:34:53.831814352Z WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2020-11-16T08:34:53.832327396Z WARNING: All illegal access operations will be denied in a future release
2020-11-16T08:34:54.979583767Z Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2020-11-16T08:34:55.137145166Z [2020-11-16T08:34:55,135][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +indy +jit [linux-x86_64]"}
2020-11-16T08:34:55.471500142Z [2020-11-16T08:34:55,471][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2020-11-16T08:34:56.043696850Z [2020-11-16T08:34:56,043][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
2020-11-16T08:34:56.043723790Z Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
2020-11-16T08:34:56.043729153Z https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
2020-11-16T08:34:56.900123735Z [2020-11-16T08:34:56,895][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@elasticsearch:9200/]}}
2020-11-16T08:34:57.365544806Z [2020-11-16T08:34:57,365][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@elasticsearch:9200/"}
2020-11-16T08:34:57.408477269Z [2020-11-16T08:34:57,408][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
2020-11-16T08:34:57.412705254Z [2020-11-16T08:34:57,410][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2020-11-16T08:34:57.556168465Z [2020-11-16T08:34:57,555][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
2020-11-16T08:34:57.557359822Z [2020-11-16T08:34:57,557][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
2020-11-16T08:34:57.661778199Z [2020-11-16T08:34:57,661][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
2020-11-16T08:34:58.693128734Z [2020-11-16T08:34:58,692][INFO ][org.reflections.Reflections] Reflections took 30 ms to scan 1 urls, producing 22 keys and 45 values 
2020-11-16T08:34:58.853347707Z [2020-11-16T08:34:58,852][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
2020-11-16T08:34:58.853385813Z ** WARNING ** You have enabled encryption but DISABLED certificate verification.
2020-11-16T08:34:58.853391576Z ** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
2020-11-16T08:34:58.898698532Z [2020-11-16T08:34:58,897][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@elasticsearch:9200/]}}
2020-11-16T08:34:58.974360067Z [2020-11-16T08:34:58,973][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@elasticsearch:9200/"}
2020-11-16T08:34:58.991679175Z [2020-11-16T08:34:58,991][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es_version=>7}
2020-11-16T08:34:58.991915099Z [2020-11-16T08:34:58,991][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
2020-11-16T08:34:59.040218347Z [2020-11-16T08:34:59,039][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["https://elasticsearch:9200"]}
2020-11-16T08:34:59.055800400Z [2020-11-16T08:34:59,055][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
2020-11-16T08:34:59.171979827Z [2020-11-16T08:34:59,171][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x6af1e75e run>"}
2020-11-16T08:34:59.769330509Z [2020-11-16T08:34:59,768][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.59}
2020-11-16T08:34:59.802126246Z [2020-11-16T08:34:59,801][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
2020-11-16T08:34:59.885412596Z [2020-11-16T08:34:59,885][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
2020-11-16T08:35:00.079522790Z [2020-11-16T08:35:00,079][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
2020-11-16T08:35:02.018848882Z [2020-11-16T08:35:02,018][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
2020-11-16T08:35:02.063938270Z [2020-11-16T08:35:02,063][INFO ][logstash.runner          ] Logstash shut down.

and logstash container keep restating again and again.
what iam doing wrong?

2

Welcome to our community! :smiley:

Can you share the command you are using to start Logstash?

3

thank you. that problem has been solved. it happened because of the user of the kibana that i used to login with. it didn't have the privilage to login. i used the super user account to log in. now i,m facing new problem when i'm trying to send docker container logs to logstash.this is my filebeat.yml file

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: container
  enabled: true
  paths:
    - '/var/lib/docker/containers/*/*.log'

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
#name: "amir_alamian_shipper"
#tags: ["logstash-test"]

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging
#fields: {project: "logstash_test", ip: "192.168.10.30"}

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  host: "https://********:5601"
  ssl.verification_mode: "none"
  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  
  #Array of hosts to connect to.
   hosts: ["localhost:9200","https://********:9200"]
   username: "elastic"
   password: "*******"
   ssl.certificate_authorities: ["/etc/pki/root/ca.crt"]
   ssl.client_authentication: "none"


  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["https://*******:5044"]
  ssl.verification_mode: "none"
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications

  # Certificate for SSL client authentication
  #sslcertificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

and this is my logstash.conf

input {
	beats {
		port => 5044
  		host => "0.0.0.0"		
	}
}

output {
	elasticsearch {
		hosts => "https://elasticsearch:9200"
    	ssl => true
		ssl_certificate_verification => true
		manage_template => false
		cacert => "/usr/share/elasticsearch/config/certificates/ca/ca.crt"
		user => 'elastic'
		password => ******
	}
}

and i get this error when setup filebeat with this command

filebeat setup -e \
                              -E output.logstash.enabled=false

error:

	ERROR	instance/beat.go:951	Exiting: 1 error: error unpacking module config: string value is not set accessing 'module'
Exiting: 1 error: error unpacking module config: string value is not set accessing 'module'
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.