I'm using this github repo *as start point. Started elasticsearch,kibana from a docker-compose file and currently trying to add logstash -f logstash.conf to work in command prompt but reaches a error message.
D:\Dev\Django\django-logs-filebeat\.docker\logstash\pipeline>logstash -f logstash.conf
to start the logstash app. Which result in the following warning message.
[2021-04-08T03:59:37,184][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
It uses Dockerfile and docker-compose.yml to run the elk stack.
I'm using docker-compose up -d to start the dockerfile
D:\Dev\Django\django-logs-filebeat>docker-compose up -d
Starting django-logs-filebeat_api_1 ... done
Recreating django-logs-filebeat_elasticsearch_1 ... done
Starting django-logs-filebeat_nginx_1 ... done
Starting django-logs-filebeat_filebeat_1 ... done
django-logs-filebeat_logstash_1 is up-to-date
django-logs-filebeat_kibana_1 is up-to-date
Then i can reach localhost:9200 for a short description and localhost:5601 for the elasticsearch and kibana. This message below is shown at localhost:9200
It's because logstash is running outside of docker so the hostname elasticsearch doesn't resolve to anything. You need to use localhost or 127.0.0.1 in ur logstash config. The error is telling u that it can't connect to elasticsearch.
[2021-04-08T04:26:24,944][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-04-08T04:26:25,376][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-04-08T04:26:25,934][INFO ][org.reflections.Reflections] Reflections took 23 ms to scan 1 urls, producing 23 keys and 47 values
[2021-04-08T04:26:26,381][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2021-04-08T04:26:26,468][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
However, nothing shows up in the kibana section -> index patterns as I thought it would ...
I'm using django and make a log file inside logs/app.log
You solved the localhost issue. so thanks legoguy. Do you have any inputs why my log file data doesn't show up in kibana? I could give you solution and start up a new question to get help with kibana logging part . What do you think is the best way to go?
So whether or not u use the native of docker version of logstash is up to u. As for the index pattern not showing up, logstash doesn't create index patterns, you'll either need to do it via the kibana interface or if ur sending data that matches the filebeat mapping have filebeat generate the initial setup.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.