Logstash 7.9.2 @Docker doesn't start -

Waxman · November 26, 2020, 5:43pm

I cannot start my logstash service on docker. Here is my config:

input {
  file {
   path => "/usr/share/logstash/messages-host"
   start_position => "beginning"
   sincedb_path => "/dev/null"
  }
  beats {
        port => 5044
        ssl => false
  }
}
output {
  elasticsearch {
    hosts => ["https://xxx:9200","https://xxxx:9200","https://xxxxx:9200"]
    user => "logstash_system"
    password => "xxx"
    ssl => true
    ssl_certificate_verification => true
    cacert => "/usr/share/logstash/config/certificates/ca/ca.crt"
}
}

Here you have my docker-compose:

version: "3"
services:
  logstash:
    image: xxx/elk/logstash/logstash:7.9.2
    container_name: uelklog01d
    environment:
      - bootstrap.memory_lock=true
      - publish-all
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
    - /srv/Docker/config/:/usr/share/logstash/config
    - ./pipeline/:/usr/share/logstash/pipeline
    - ./testdata/messages:/usr/share/logstash/messages-host
    - /srv/Docker/logstash_data:/usr/share/logstash/data
    - /srv/Docker/logs/:/usr/share/logstash/logs
 ports:
     - 9600:9600
     - 5044:5044
    command: /usr/share/logstash/bin/logstash -f /usr/share/logstash/config/logstash.conf
    network_mode: "host"

my logstash.yml:

node.name: uelklog01d
path.data: /usr/share/logstash/data
http.host: 0.0.0.0
http.port: 9600-9700
log.level: debug
path.logs: /usr/share/logstash/logs
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: "xxx"
xpack.monitoring.elasticsearch.hosts: ["https://xxx:9200", "https://xxxx:9200", "https://xxxxx:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/usr/share/logstash/config/certificates/ca/ca.crt" ]
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate

and finally these are logs from console:

uelklog01d  | [2020-11-26T17:17:04,601][DEBUG][logstash.runner          ] *path.config: "/usr/share/logstash/config/logstash.conf"
uelklog01d  | [2020-11-26T17:17:04,603][DEBUG][logstash.runner          ] path.data: "/usr/share/logstash/data"
uelklog01d  | [2020-11-26T17:17:04,604][DEBUG][logstash.runner          ] modules.cli: []
uelklog01d  | [2020-11-26T17:17:04,605][DEBUG][logstash.runner          ] modules: []
uelklog01d  | [2020-11-26T17:17:04,607][DEBUG][logstash.runner          ] modules_list: []
uelklog01d  | [2020-11-26T17:17:04,610][DEBUG][logstash.runner          ] modules_variable_list: []
uelklog01d  | [2020-11-26T17:17:04,611][DEBUG][logstash.runner          ] modules_setup: false
uelklog01d  | [2020-11-26T17:17:04,613][DEBUG][logstash.runner          ] config.test_and_exit: false
uelklog01d  | [2020-11-26T17:17:04,614][DEBUG][logstash.runner          ] config.reload.automatic: false
uelklog01d  | [2020-11-26T17:17:04,615][DEBUG][logstash.runner          ] config.reload.interval: #<LogStash::Util::TimeValue:0x47a1b2f7 @duration=3, @time_unit=:second>
uelklog01d  | [2020-11-26T17:17:04,616][DEBUG][logstash.runner          ] config.support_escapes: false
uelklog01d  | [2020-11-26T17:17:04,617][DEBUG][logstash.runner          ] config.field_reference.parser: "STRICT"
uelklog01d  | [2020-11-26T17:17:04,619][DEBUG][logstash.runner          ] metric.collect: true
uelklog01d  | [2020-11-26T17:17:04,620][DEBUG][logstash.runner          ] pipeline.id: "main"
uelklog01d  | [2020-11-26T17:17:04,621][DEBUG][logstash.runner          ] pipeline.system: false
uelklog01d  | [2020-11-26T17:17:04,622][DEBUG][logstash.runner          ] pipeline.workers: 4
uelklog01d  | [2020-11-26T17:17:04,623][DEBUG][logstash.runner          ] pipeline.batch.size: 125
uelklog01d  | [2020-11-26T17:17:04,624][DEBUG][logstash.runner          ] pipeline.batch.delay: 50
uelklog01d  | [2020-11-26T17:17:04,625][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
uelklog01d  | [2020-11-26T17:17:04,626][DEBUG][logstash.runner          ] pipeline.java_execution: true
uelklog01d  | [2020-11-26T17:17:04,627][DEBUG][logstash.runner          ] pipeline.reloadable: true
uelklog01d  | [2020-11-26T17:17:04,628][DEBUG][logstash.runner          ] pipeline.plugin_classloaders: false
uelklog01d  | [2020-11-26T17:17:04,629][DEBUG][logstash.runner          ] pipeline.separate_logs: false
uelklog01d  | [2020-11-26T17:17:04,630][DEBUG][logstash.runner          ] pipeline.ordered: "auto"
uelklog01d  | [2020-11-26T17:17:04,631][DEBUG][logstash.runner          ] path.plugins: []
uelklog01d  | [2020-11-26T17:17:04,632][DEBUG][logstash.runner          ] config.debug: false
uelklog01d  | [2020-11-26T17:17:04,633][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
uelklog01d  | [2020-11-26T17:17:04,633][DEBUG][logstash.runner          ] version: false
uelklog01d  | [2020-11-26T17:17:04,634][DEBUG][logstash.runner          ] help: false
uelklog01d  | [2020-11-26T17:17:04,635][DEBUG][logstash.runner          ] log.format: "plain"
uelklog01d  | [2020-11-26T17:17:04,636][DEBUG][logstash.runner          ] http.enabled: true
uelklog01d  | [2020-11-26T17:17:04,637][DEBUG][logstash.runner          ] *http.host: "0.0.0.0" (default: "127.0.0.1")
uelklog01d  | [2020-11-26T17:17:04,638][DEBUG][logstash.runner          ] http.port: 9600..9700
uelklog01d  | [2020-11-26T17:17:04,639][DEBUG][logstash.runner          ] http.environment: "production"
uelklog01d  | [2020-11-26T17:17:04,640][DEBUG][logstash.runner          ] queue.type: "memory"
uelklog01d  | [2020-11-26T17:17:04,641][DEBUG][logstash.runner          ] queue.drain: false
uelklog01d  | [2020-11-26T17:17:04,642][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
uelklog01d  | [2020-11-26T17:17:04,643][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
uelklog01d  | [2020-11-26T17:17:04,644][DEBUG][logstash.runner          ] queue.max_events: 0
uelklog01d  | [2020-11-26T17:17:04,644][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
uelklog01d  | [2020-11-26T17:17:06,138][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '403' contacting Elasticsearch at URL 'https://xxxxx:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>
uelklog01d  |   setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:50
uelklog01d  | [2020-11-26T17:17:06,328][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in head'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.6.2-java/lib/logstash/outputs/elasticsearch/common.rb:50:in `block in setup_after_successful_connection'"]}
uelklog01d  | [2020-11-26T17:17:06,333][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
uelklog01d exited with code 1

Any help would be appreciate.

Badger · November 26, 2020, 7:18pm

When it tries to setup ilm it sends a request to elasticsearch to determine whether the rollover index exists. That is getting a 403 status. Do the elasticsearch logs tell you anything?

Waxman · November 27, 2020, 8:05am

I found only this:

stacktrace": ["org.elasticsearch.ElasticsearchSecurityException: action [indices:admin/get] is unauthorized for user [logstash_system],

This is strange cause user logstash_system has got privileges to the cluster:

curl --cacert /pwd/certificates/ca/ca.crt -ku logstash_system https://xxx:9200/_cluster/health?pretty
Enter host password for user logstash_system:
{
  "cluster_name" : "es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 60,
  "active_shards" : 120,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Waxman · November 27, 2020, 9:42am

The funny thing: when I change the user from logstash_system to elastic then everything went ok. What' s the casue? If there is a need to add some privileges to the logstash_system ?

Badger · November 27, 2020, 1:47pm

Not sure, that's really an elasticsearch question, not a logstash question.

Waxman · November 27, 2020, 2:10pm

I will try this advice:https://www.elastic.co/guide/en/logstash/current/ls-security.html

system · December 25, 2020, 2:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.