Resolving Message Queue Clogs in Elasticsearch with Informatica C360

Elastic Stack Elasticsearch
1

Hello Team,

In our organization, we use Informatica C360 for Customer MDM, alongside Elastic Search to implement the Smart Search capability that Informatica provides. While we appreciate the optimized searching techniques from ES, we are currently facing an issue affecting the "Search after Create" functionality from our external applications, which is crucial from a business standpoint.

We are collaborating with the Informatica GCS team to address this issue, and we would also like to seek your expertise for a potential solution. Working with GCS, we have narrowed it down to the message-count/messages-added that show the number of documents waiting for indexing. Despite the count not even crossing four digits, we are encountering this issue from external applications. The issue resolves itself whenever we flush out the SSS message queue.

It's also important to note that the consumer-count is set to 15, which is static. I wonder if there is a way to make it scale up dynamically so that it can process the records as they come in, preventing a clog when the messages-added increases.

Looking forward to hearing back with a potential solutions.

Thank you,
Selva

2

Hello and welcome,

It is not clear what is your specific issue with Elasticsearch.

From what you describe the issue is with a tool that uses Elasticsearch as part of its backend, you need to check with the vendor of the tool, it is not possible to know how they are using elasticseach and what they are doing with it, so it is pretty hard to provide any feedback.

3

Hi Leandrojmp - Thank you for sharing your thoughts.