# Logstash is not reading data in Docker **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [April 10, 2023, 3:01pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666 "2023-04-10T15:01:12Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![vvsenthil](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vvsenthil](https://discuss.elastic.co/u/vvsenthil)\ **Post date:** [April 10, 2023, 3:01pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/1 "2023-04-10T15:01:12Z")
Hi, Someone would you be able to help me on setting up the logstah in docker. I am able to setup and push the message to Elasticsearch without docker. But if i move the logstah to docker i am not able to push the message. Would be able to help me to setup the environment. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 10, 2023, 3:18pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/2 "2023-04-10T15:18:09Z")
You need to provide more context and share your configurations. What you want to read? Where is the data you want to read? ---
**Author:** ![vvsenthil](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vvsenthil](https://discuss.elastic.co/u/vvsenthil)\ **Post date:** [April 10, 2023, 3:33pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/3 "2023-04-10T15:33:41Z")
Hi, Please find the docker compose file below. ```auto version: "3.0" services: elasticsearch: container_name: elastic-container image: docker.elastic.co/elasticsearch/elasticsearch:7.13.4 environment: - xpack.security.enabled=false - "discovery.type=single-node" networks: - ek-net ports: - 9200:9200 kibana: container_name: kibana-container image: docker.elastic.co/kibana/kibana:7.13.4 environment: - ELASTICSEARCH_HOSTS=http://elastic-container:9200 networks: - ek-net depends_on: - elasticsearch ports: - 5601:5601 networks: ek-net: driver: bridge ``` and my logstash.conf file as below: ```auto input { file { codec => "json" path => "/usr/share/logstash/data/*.json" start_position => beginning } } filter { mutate{ rename => { "data_lat" => "latitude" } rename => { "data_long" => "longitude" } convert => {"latitude" => "float"} convert => {"longitude" => "float"} add_field => ["location","%{latitude},%{longitude}"] } } output { elasticsearch { hosts => ["localhost:9200"] index => "customer_index" } #stdout { codec => rubydebug } } ``` I have placed the json file in /usr/share/logstash/data docker location. Could you please suggest whether i am following the correct approach or do i need to change anything? ---
**Author:** ![vvsenthil](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vvsenthil](https://discuss.elastic.co/u/vvsenthil)\ **Post date:** [April 12, 2023, 7:52am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/4 "2023-04-12T07:52:17Z")
Any idea ? ---
**Author:** ![TimBosman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timbosman/32/84120_2.png) [@TimBosman](https://discuss.elastic.co/u/TimBosman)\ **Post date:** [April 12, 2023, 8:02am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/5 "2023-04-12T08:02:42Z")
Hey, How I do it is mounting a pipelines.yml on `/usr/share/logstash/config/pipelines.yml` and mount my pipeline in `/usr/share/logstash/pipeline/`. My pipelines.yml looks like this: ```auto - pipeline.id: new path.config: "/usr/share/logstash/pipeline/new.conf" queue.type: persisted ``` More information on the pipelines.yml can be found [here](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) I hope this helps, Tim ---
**Author:** ![vvsenthil](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vvsenthil](https://discuss.elastic.co/u/vvsenthil)\ **Post date:** [April 12, 2023, 9:42pm UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/6 "2023-04-12T21:42:10Z")
Hi, I have done this changes, but no luck. Getting the below error. Error in Logstash: ```auto 2023-04-12 22:27:39 [2023-04-12T21:27:39,452][WARN][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml 2023-04-12 22:27:39 [2023-04-12T21:27:39,458][WARN][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release. 2023-04-12 22:27:39 Please configure Metricbeat to monitor Logstash. Documentation can be found at: 2023-04-12 22:27:39 https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html 2023-04-12 22:27:40 [2023-04-12T21:27:40,019][WARN][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. 2023-04-12 22:27:40 [2023-04-12T21:27:40,114][WARN][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. 2023-04-12 22:27:40 [2023-04-12T21:27:40,538][INFO][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} 2023-04-12 22:27:40 [2023-04-12T21:27:40,729][WARN][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:27:40 [2023-04-12T21:27:40,790][WARN][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused) {:url=>http://localhost:9200/, :error_message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} 2023-04-12 22:27:40 [2023-04-12T21:27:40,805][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:27:40 [2023-04-12T21:27:40,880][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster. 2023-04-12 22:27:41 [2023-04-12T21:27:41,130][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} 2023-04-12 22:27:41 [2023-04-12T21:27:41,920][INFO][org.reflections.Reflections] Reflections took 102 ms to scan 1 urls, producing 119 keys and 417 values 2023-04-12 22:27:42 [2023-04-12T21:27:42,646][WARN][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. 2023-04-12 22:27:42 [2023-04-12T21:27:42,699][WARN][deprecation.logstash.inputs.file] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. 2023-04-12 22:27:42 [2023-04-12T21:27:42,774][WARN][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. 2023-04-12 22:27:42 [2023-04-12T21:27:42,818][WARN][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. 2023-04-12 22:27:43 [2023-04-12T21:27:43,130][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} 2023-04-12 22:27:43 [2023-04-12T21:27:43,193][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} 2023-04-12 22:27:43 [2023-04-12T21:27:43,217][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:27:43 [2023-04-12T21:27:43,358][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/logstash_dir/logstash.conf"], :thread=>"#"} 2023-04-12 22:27:44 [2023-04-12T21:27:44,353][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>0.99} 2023-04-12 22:27:44 [2023-04-12T21:27:44,437][INFO][logstash.inputs.file][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_6b6a587863045b058bf1fabc0c50dd07", :path=>["/logstash_dir/inlog.log"]} 2023-04-12 22:27:44 [2023-04-12T21:27:44,481][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} 2023-04-12 22:27:44 [2023-04-12T21:27:44,586][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} 2023-04-12 22:27:44 [2023-04-12T21:27:44,594][INFO][filewatch.observingtail][main][4520fe864e6ba9b0cb557b754486d14415cc8b80d7b6cd26b876efe9239a082b] START, creating Discoverer, Watch with file and sincedb collections 2023-04-12 22:27:48 [2023-04-12T21:27:48,232][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:27:53 [2023-04-12T21:27:53,244][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:27:58 [2023-04-12T21:27:58,255][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:28:03 [2023-04-12T21:28:03,266][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:28:08 [2023-04-12T21:28:08,276][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:28:10 [2023-04-12T21:28:10,777][WARN][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} 2023-04-12 22:28:10 [2023-04-12T21:28:10,850][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"} 2023-04-12 22:28:13 [2023-04-12T21:28:13,287][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [April 13, 2023, 5:00am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/7 "2023-04-13T05:00:16Z")
> [@vvsenthil](#): > > ```auto > 2023-04-12 22:27:40 [2023-04-12T21:27:40,805][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"} > 2023-04-12 22:27:40 [2023-04-12T21:27:40,880][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster. > 20 > > ``` Your ES is not reachable to LS. Please check: - firewall port 9200 - `network.host` settings in elasticsearch.yml, should be set to locahost or 0.0.0.0 - availability ES by curl ---
**Author:** ![vvsenthil](https://avatars.discourse-cdn.com/v4/letter/v/f6c823/32.png) [@vvsenthil](https://discuss.elastic.co/u/vvsenthil)\ **Post date:** [April 13, 2023, 8:08am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/8 "2023-04-13T08:08:14Z")
Hi Rios thanks for your reply. Yes you are correct. I followed below solution and able to push the data to Elasticsearch. > [@Logstash docker image doesn't connect to elasticsearch image](https://discuss.elastic.co/t/logstash-docker-image-doesnt-connect-to-elasticsearch-image/113691/3): > > It works perfectly well now! Thank you very much for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 11, 2023, 8:08am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-data-in-docker/329666/9 "2023-05-11T08:08:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.