Pull data from Elastic Enterprise Search to Kafka Topic

[albertopneto (Alberto Ribeiro)]

Hi guys,

I'm doing some experiments trying to pull data from Elasticsearch and pushing to Kafka.

The source connector I'm using from Kafka Connect requires an index name and in this case with Elastic Enterprise Search I don't have, I only have documents, also, the source connector doesn't support Query, hence I can't pass the query to the connector in order to get the documents...

Is there any alternative for kafka connect ? Do you know any tool/framework that allows me to pull data from Elastic Enterprise Search and push to Kafka ?

Thank you so much in advance.

[Irina_Truong]

Hi @albertopneto ,

You always have the backing Elasticsearch index that contains your documents. The index name would start with .ent-search-engine-documents- prefix.

If you're using App Search, the index would be named .ent-search-engine-documents-{ENGINE_NAME}.

If you're using Workplace Search, the index name would be .ent-search-engine-documents-{CONTENT_SOURCE_TYPE}-{CONTENT_SOURCE_ID}.

You can list all indices in your Elasticsearch that start with the prefix and find the one you need. Example:

curl -XGET 'http://localhost:9200/_cat/indices/.ent-search-engine-documents*'.

[albertopneto (Alberto Ribeiro)]

Thank you so much for the quick response Irina... I was able to get all the index from the UI (API Console) but I'm not able to call the same from postman or curl command.

Below the url I'm trying to hit:

https://my-enterprise-search-deployment.ent.europe-west1.gcp.cloud.es.io/.ent-search-engine-documents-[engine-name]/_doc/[docId]

Even the one to get all the indexes I can't hit from curl command, trying as follows:
https://my-enterprise-search-deployment.ent.europe-west1.gcp.cloud.es.io/_cat/indices/.ent-search-engine-documents*

Am I missing anything? is the api in a different address ?

Thank you so much again.

[Irina_Truong]

@albertopneto The URL that you're using has .ent. in it - this is the Enterprise Search URL. The _cat API to list indices is present in Elasticsearch, not Enterprise Search. You'd have to use the Elasticsearch endpoint URL instead.

[albertopneto (Alberto Ribeiro)]

Hi Irina, thank you so much ...

A final question if you allow me... is the Elasticsearch API publicly accessible ?

I can run any request from the API Console (UI) using my login credentials but I can't make any request to the Elasticsearch endpoint from Postman.

Best regards,
Alberto Ribeiro

[albertopneto (Alberto Ribeiro)]

Hi Irina,

I was able to make that to work, calls to Elasticsearch endpoints... It worked in two different ways.

1. I created an Api Key using the /_security/api_key API from the console, then, I get the base64 form of ApiKeyId:ApiKey and set it to the header...
2. Created an user in Elasticsearch and give few permissions to allow the user to request indexes OR documents, then, set the user and pass into Postman Authorization section (Basic type).

All good,

Again, thank you so much
Have a nice weekend

[Serena_Chou]

@albertopneto hey Alberto, glad to see that you're on track to getting this done! I just wanted to surface along with Irina's excellent answers that in later versions of Enterprise Search we do provide additional ways to great Elasticsearch indices that would be prefixed search-. Would you be interested in giving us some feedback as to what you're planning to do with the content you're pushing into kafka connect?

[albertopneto (Alberto Ribeiro)]

Hi @Serena_Chou ,

Basically, I'm building a pipeline using Kafka Connect to move data from Elasticsearch to Redis Cache, the data I'm moving is going to be used by a microservice.
The reason I'm pushing the data to Redis is to avoid making http requests to Elasticsearch every time I need to get the data, with the data into the cache I can easily access it.

does that make any sense ?

Best regards,
Alberto Ribeiro