Forward logs from Kafka to Elastic

[Deepika1]

Hello,

I'm trying to forward logs from Kafka machine to Elasticsearch machine using kafka elastic connect referring below document

Kafka Connect Elasticsearch: Kafka Connector Consuming & Indexing - Sematext

configurations are:
name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=logs
topic.index.map=logs:logs_index
connection.url=http://192.168.1.18:9200 // elasticsearch ip
type.name=log
key.ignore=true
schema.ignore=true

My kafka cluster ip address is : 192.168.1.9 , 192.168.1.10
However if i'm installing elasticsearch on same machine as kafka and changing connection.url=http://localhost:9200 i'm able to view index created without any docs in it

Require your assistance
Thanks in advance

[warkolm (Mark Walkom)]

Welcome to our community!

We may not be able to help here as it sounds like kafka isn't sending the data to Elasticsearch, you may need to ask somewhere more kafka specific.

[Christian_Dahlqvist]

Note that this document is quite old. One thing that has changed in Elasticsearch is that document types have been deprecated and removed so it is possible that type.name=log is causing problems if you are using a recent version of Elasticsearch. Try removing this or change it to _doc. I am not familiar with this connector and it is not supported here, so you may need to reach out to the maintainers or the Kafka community for guidance and compatibility information.

[Deepika1]

Thank you for your response. I tried with given solution, but it isn't working.
Do you have any more suggestion?