Forward logs from Kafka to Elastic

Elastic Stack Elasticsearch
1

Hello,

I'm trying to forward logs from Kafka machine to Elasticsearch machine using kafka elastic connect referring below document

configurations are:
name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=logs
topic.index.map=logs:logs_index
connection.url=http://192.168.1.18:9200 // elasticsearch ip
type.name=log
key.ignore=true
schema.ignore=true

My kafka cluster ip address is : 192.168.1.9 , 192.168.1.10
However if i'm installing elasticsearch on same machine as kafka and changing connection.url=http://localhost:9200 i'm able to view index created without any docs in it

Require your assistance
Thanks in advance

2

Welcome to our community! :smiley:

We may not be able to help here as it sounds like kafka isn't sending the data to Elasticsearch, you may need to ask somewhere more kafka specific.

3

Note that this document is quite old. One thing that has changed in Elasticsearch is that document types have been deprecated and removed so it is possible that type.name=log is causing problems if you are using a recent version of Elasticsearch. Try removing this or change it to _doc. I am not familiar with this connector and it is not supported here, so you may need to reach out to the maintainers or the Kafka community for guidance and compatibility information.

4

Thank you for your response. I tried with given solution, but it isn't working.
Do you have any more suggestion?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.