Logstash not able to pass data to elasticsearch

Elastic Stack Elasticsearch
1

I have setup Logstash server 1-> Kafka -> Logstash server 2-> Elasticsearch -> Kibana setup. Below is Logstash server 2 configuration files.

  1. 03-logstash-logs-kafka-consumer.conf

input {
kafka {
zk_connect => 'zk_netaddress:2181'
topic_id => 'logstash_logs'
codec => "json"
}
}
output{
stdout{}
}

  1. 30-elasticsearch-output.conf

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Though Logs are travelling from Logstash server 1 to Logstash server 2 through Kafka and Logstash server 2 can also output it to /var/log/logstash/logstash.stdout file but It is not able to output to elasticsearch configure with it. I have checked all services are running well and not Exception in logs of any service. Please post your suggestion. Thanks a lot in advance.

2

Are you're able to send this to your ES instance:

curl -XPUT "http://hostname:9200/test/test/1" -d '{}'

If you're able to do this, then the issue is on the Logstash side.

3

I am able to do curl -XPUT "http://hostname:9200/test/test/1" -d '{}' . Still facing issue same issue mentioned in Post.

4

Hi ,

So if the issue is at logstash ,we do not see any related suspicious log entries in logstash logs.
I enabled debug logs for logstash but that too did not help.
So what could be the reasons behind this. Does logstash needs any specific configuration when use with ELK ?

Regards,
Vinod

5

Hi gaurav,

I am facing the same issue. Any resolution to this?