I m running logstash in a container

Elastic Stack Logstash
1

Why does this runs pipelines.yml. while passing the configuration using -f.
I use startup.sh which in turn calls the /usr/share/logstash/bin/logstash -f /usr/share/logstash/conf.d/ -w 2
Within the container I see 2 process running
logstash 1 0 0 03:46 ? 00:00:00 /bin/bash /opt/config/startup.sh
logstash 15 1 15 03:46 ? 00:01:17 /usr/share/logstash/jdk/bin/java -Xms2G -Xmx8G -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedy

But I see in the logs it still runs pipelines.yml.
Please find the logs attached

[2023-09-08T03:47:38,142][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/, :path=>"/"}
[2023-09-08T03:47:38,171][WARN ][logstash.outputs.amazonelasticsearch][main] Restored connection to ES instance {:url=>"https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/"}
[2023-09-08T03:47:38,267][INFO ][logstash.outputs.amazonelasticsearch][main] ES Output version determined {:es_version=>7}
[2023-09-08T03:47:38,267][WARN ][logstash.outputs.amazonelasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2023-09-08T03:47:48,354][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/usr/share/logstash/conf.d/logstash-elasticsearch-output.conf", "/usr/share/logstash/conf.d/logstash-filter.conf", "/usr/share/logstash/conf.d/logstash-fingerprint.conf", "/usr/share/logstash/conf.d/logstash-input.conf"], :thread=>"#<Thread:0x634300d8@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-09-08T03:47:51,248][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2023-09-08T03:48:21,758][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}
[2023-09-08T03:48:21,759][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}

Please help

2

This log line appears to show logstash is running the pipelines from the directory you provided.

I do not see where you think it is running pipelines.yml

You also have an issue connecting to elasticsearch

3

Thank you @stephenb for helping.
In the logs, at the beginning it contacts the right host

[2023-09-08T03:47:38,142][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/, :path=>"/"}
[2023-09-08T03:47:38,171][WARN ][logstash.outputs.amazonelasticsearch][main] Restored connection to ES instance {:url=>"https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/"}
[2023-09-08T03:47:38,267][INFO ][logstash.outputs.amazonelasticsearch][main] ES Output version determined {:es_version=>7}

But from where does this below one come from.

[2023-09-08T03:48:21,759][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}

4

You would need to share all your *.conf files

They get concatenated together perhaps you have more than one output defined

You will need to share much more for help.

5

Thanks again @stephenb for helping.
This is the wrapper/ start up script which i run.

    #!/bin/bash

    mkdir /usr/share/logstash/conf.d/
    cp /opt/config/logstash-input.conf /usr/share/logstash/conf.d/
    cp /opt/config/logstash-filter.conf /usr/share/logstash/conf.d/
    cp /opt/config/logstash-fingerprint.conf /usr/share/logstash/conf.d/
    cp /opt/config/logstash-elk-output.conf /usr/share/logstash/conf.d/
    /usr/share/logstash/bin/logstash -f /usr/share/logstash/conf.d/ -w 2 
    
   I have checked all the files in directory  /usr/share/logstash/conf.d/ to see why it is connecting to {:url=>"http://elasticsearch:9200/",}

please let me know if you need more information

6

I do not know what the contents of any of those files are....so I can't help.

Please show any files that have an output section.

It seems like perhaps you did not create the files?

Also share your logstash.yml file

7

Thanks @stephenb.

Contents of output file from /usr/share/logstash/conf.d

cat logstash-elasticsearch-output.conf

cat logstash-elasticsearch-output.conf

output {
  if [type] =~ "default" {
    amazon_es {
      hosts => ["https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/"]
      template_overwrite => true
      index => "%{service}-%{+YYYY.MM.dd}"
      region => ["us-test-2"]
      document_id => "%{fingerprint}"
    }
  } else if [type] =~ "json" {
    amazon_es {
      hosts => ["https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/"]
      template_overwrite => true
      index => "kong-%{+YYYY.MM.dd}"
      region => ["us-test-2"]
      document_id => "%{fingerprint}"       
    }
  } else if [type] == "dlp-messages" or [type] == "dlp-stats" or  [type] == "dlp2-messages" or [type] == "dlp2-stats"  {
    amazon_es {
      hosts => ["https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/"]
      template_overwrite => true
      index => "%{type}-%{[@metadata][subtype]}-%{+YYYY.MM.dd}"
      region => ["us-test-2"]
      document_id => "%{fingerprint}"
    }
  } 

  
  else {
    amazon_es {
      hosts => ["https://test-logstash-biihs64z73je2wmv6xh5oqwwxm.us-test-2.es.amazonaws.com:443/"]
      template_overwrite => true
      index => "%{type}-%{+YYYY.MM.dd}"
      region => ["us-test-2"] 
      document_id => "%{fingerprint}"     
    }
  }
}

cat /usr/share/logstash/config/logstash.yml 
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
8

There is why it's trying to connect to that elasticsearch host logstash is trying to send its own monitoring data you can comment that out... or change it to the hosts in the output section.

Seems like perhaps you should read a bit of the logstash documentation.

9

Thank you @stephenb. I appreciate your help.
The only issue is that logs are not getting logged to the directory logs. I dont see any logstash-plain.log or such. Please help

10

@stephenb As per the documentation.
Logstash Docker containers do not create log files by default. They log to standard output.

Custom setting changes is required for writing logs.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.