Hello everyone, I am having an error when running the logstash on the docker.
I'm trying to run logstash on the docker with the following command:
command:
docker run --rm -it -v /usr/share/logstash/pipeline/logstash.conf:/home/jonas/elastic-stack/logstash-7.12.1/config/logstash-sample.conf docker.elastic.co/logstash/logstash:7.13.0
the configuration file is the same as I use with logstash without the docker, in this case it works.
Configuration file:
input {
beats {
port => 5044
}
}
filter {
prune {
whitelist_names => [ "message" ]
}
}
output {
s3{
access_key_id => "XXXXXXXXXXX"
secret_access_key => "YYYYYYYYYYYYYYYYYYYYY"
region => "sa-east-1"
bucket => "ortiz-aws-bucket"
size_file => 5000
time_file => 2
codec => "json_lines"
canned_acl => "private"
}
stdout{}
}
Error log:
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2021-06-01T11:53:47,742][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2021-06-01T11:53:47,750][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.0", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[2021-06-01T11:53:47,774][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2021-06-01T11:53:47,788][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2021-06-01T11:53:48,256][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"8d1e7f3c-96b3-414c-a92e-5a0923cb7999", :path=>"/usr/share/logstash/data/uuid"}
[2021-06-01T11:53:48,720][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2021-06-01T11:53:48,722][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at:
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[2021-06-01T11:53:49,106][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-06-01T11:53:49,498][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2021-06-01T11:53:59,651][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
[2021-06-01T11:53:59,694][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2021-06-01T11:53:59,702][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2021-06-01T11:53:59,740][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2021-06-01T11:54:00,126][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-01T11:54:00,598][INFO ][org.reflections.Reflections] Reflections took 52 ms to scan 1 urls, producing 24 keys and 48 values
[2021-06-01T11:54:01,240][WARN ][deprecation.logstash.inputs.beats] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-06-01T11:54:01,556][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x43e0b7f9 run>"}
[2021-06-01T11:54:02,255][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.7}
[2021-06-01T11:54:02,282][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2021-06-01T11:54:02,299][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-06-01T11:54:02,375][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-06-01T11:54:02,396][INFO ][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] Starting server on port: 5044
[2021-06-01T11:54:29,724][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2021-06-01T11:54:40,140][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
Apparently my config file is not being used?
Someone could help me ?