Hi, Someone would you be able to help me on setting up the logstah in docker. I am able to setup and push the message to Elasticsearch without docker. But if i move the logstah to docker i am not able to push the message. Would be able to help me to setup the environment.
You need to provide more context and share your configurations.
What you want to read? Where is the data you want to read?
Hi, Please find the docker compose file below.
version: "3.0"
services:
elasticsearch:
container_name: elastic-container
image: docker.elastic.co/elasticsearch/elasticsearch:7.13.4
environment:
- xpack.security.enabled=false
- "discovery.type=single-node"
networks:
- ek-net
ports:
- 9200:9200
kibana:
container_name: kibana-container
image: docker.elastic.co/kibana/kibana:7.13.4
environment:
- ELASTICSEARCH_HOSTS=http://elastic-container:9200
networks:
- ek-net
depends_on:
- elasticsearch
ports:
- 5601:5601
networks:
ek-net:
driver: bridge
and my logstash.conf file as below:
input {
file {
codec => "json"
path => "/usr/share/logstash/data/*.json"
start_position => beginning
}
}
filter {
mutate{
rename => { "data_lat" => "latitude" }
rename => { "data_long" => "longitude" }
convert => {"latitude" => "float"}
convert => {"longitude" => "float"}
add_field => ["location","%{latitude},%{longitude}"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "customer_index"
}
#stdout { codec => rubydebug }
}
I have placed the json file in /usr/share/logstash/data docker location.
Could you please suggest whether i am following the correct approach or do i need to change anything?
Any idea ?
Hey,
How I do it is mounting a pipelines.yml on /usr/share/logstash/config/pipelines.yml and mount my pipeline in /usr/share/logstash/pipeline/.
My pipelines.yml looks like this:
- pipeline.id: new
path.config: "/usr/share/logstash/pipeline/new.conf"
queue.type: persisted
More information on the pipelines.yml can be found here
I hope this helps,
Tim
Hi, I have done this changes, but no luck.
Getting the below error.
Error in Logstash:
2023-04-12 22:27:39 [2023-04-12T21:27:39,452][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
2023-04-12 22:27:39 [2023-04-12T21:27:39,458][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release.
2023-04-12 22:27:39 Please configure Metricbeat to monitor Logstash. Documentation can be found at:
2023-04-12 22:27:39 https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
2023-04-12 22:27:40 [2023-04-12T21:27:40,019][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
2023-04-12 22:27:40 [2023-04-12T21:27:40,114][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
2023-04-12 22:27:40 [2023-04-12T21:27:40,538][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
2023-04-12 22:27:40 [2023-04-12T21:27:40,729][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:27:40 [2023-04-12T21:27:40,790][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused) {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2023-04-12 22:27:40 [2023-04-12T21:27:40,805][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:27:40 [2023-04-12T21:27:40,880][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2023-04-12 22:27:41 [2023-04-12T21:27:41,130][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2023-04-12 22:27:41 [2023-04-12T21:27:41,920][INFO ][org.reflections.Reflections] Reflections took 102 ms to scan 1 urls, producing 119 keys and 417 values
2023-04-12 22:27:42 [2023-04-12T21:27:42,646][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
2023-04-12 22:27:42 [2023-04-12T21:27:42,699][WARN ][deprecation.logstash.inputs.file] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
2023-04-12 22:27:42 [2023-04-12T21:27:42,774][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
2023-04-12 22:27:42 [2023-04-12T21:27:42,818][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
2023-04-12 22:27:43 [2023-04-12T21:27:43,130][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
2023-04-12 22:27:43 [2023-04-12T21:27:43,193][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
2023-04-12 22:27:43 [2023-04-12T21:27:43,217][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:27:43 [2023-04-12T21:27:43,358][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/logstash_dir/logstash.conf"], :thread=>"#<Thread:0x7ab8382b run>"}
2023-04-12 22:27:44 [2023-04-12T21:27:44,353][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.99}
2023-04-12 22:27:44 [2023-04-12T21:27:44,437][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_6b6a587863045b058bf1fabc0c50dd07", :path=>["/logstash_dir/inlog.log"]}
2023-04-12 22:27:44 [2023-04-12T21:27:44,481][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
2023-04-12 22:27:44 [2023-04-12T21:27:44,586][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
2023-04-12 22:27:44 [2023-04-12T21:27:44,594][INFO ][filewatch.observingtail ][main][4520fe864e6ba9b0cb557b754486d14415cc8b80d7b6cd26b876efe9239a082b] START, creating Discoverer, Watch with file and sincedb collections
2023-04-12 22:27:48 [2023-04-12T21:27:48,232][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:27:53 [2023-04-12T21:27:53,244][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:27:58 [2023-04-12T21:27:58,255][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:28:03 [2023-04-12T21:28:03,266][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:28:08 [2023-04-12T21:28:08,276][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:28:10 [2023-04-12T21:28:10,777][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
2023-04-12 22:28:10 [2023-04-12T21:28:10,850][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2023-04-12 22:28:13 [2023-04-12T21:28:13,287][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
Your ES is not reachable to LS. Please check:
- firewall port 9200
network.hostsettings in elasticsearch.yml, should be set to locahost or 0.0.0.0- availability ES by curl
Hi Rios thanks for your reply. Yes you are correct. I followed below solution and able to push the data to Elasticsearch.
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.