Request for Assistance – Application and ELK Data Count Mismatch

Dear Team,

We are currently experiencing a significant data count mismatch between the API Gateway and ELK. We would like to highlight the issue, along with the observations and validations performed so far.

Issue Background:

Initially, when we cross-verified the data count between the API Gateway and ELK, the counts were matching correctly. At that time, only 12 APIs were live.

Subsequently, additional APIs were onboarded into ELK. However, the data count was not cross-verified immediately after onboarding. During our recent validation, we observed a significant mismatch of approximately 90% between the API Gateway and ELK data counts.

Observations and Validations Performed:

Elasticsearch Logs: We reviewed the Elasticsearch logs to identify any errors or failures related to data ingestion. However, no relevant errors were identified during the investigation.

Elasticsearch Thread Pool: We checked the Elasticsearch write thread pool statistics to identify any rejected requests. However, no rejected requests were observed during the validation.

API Gateway Logs: While reviewing the Gateway logs, we identified the following error:

"Event drop for destination.elasticsearch due to slow/unreachable destination."

This indicates that some events are being dropped at the Gateway end while forwarding data to the destination Elasticsearch.

Firewall Validation: We also coordinated with the firewall team to verify whether any packets were being dropped during communication between the source and destination. During the validation, client resets were observed.

Additional Application Validation: As part of further investigation, we cross-verified the data count for another application whose logs are being ingested through Filebeat. The data count for this application is matching correctly. However, the API Gateway data, which is being forwarded from Elasticsearch to Elasticsearch, continues to show a significant mismatch.

Request for Further Investigation:

Based on the above observations, we request your assistance in identifying the root cause of the issue and providing an appropriate resolution.

Kindly help us investigate the following:

The reason behind the event drops at the API Gateway end.

The cause of the slow/unreachable destination error and client resets.

Whether there are any limitations or bottlenecks in the Elasticsearch-to-Elasticsearch data forwarding mechanism, particularly after the onboarding of additional APIs.

Any configuration changes or optimizations required to ensure that all events are successfully delivered to the destination Elasticsearch without data loss.

Your assistance in identifying the root cause and resolving this issue would be highly appreciated, as the data count mismatch is impacting the accuracy of monitoring and reporting in ELK.

Looking forward to your response and support

The Gateway message indicates backpressure at the destination rather than Elasticsearch rejecting writes. I would correlate the drop timestamps with destination bulk latency, client resets, and Gateway send-queue depth. Compare the API Gateway pipeline with the working Filebeat path: bulk size, concurrency, request timeouts, retry/backoff, and persistent buffering. Also check destination index refresh interval, shard count, mapping conflicts, and ingest pipeline failures, since these can slow bulk responses without appearing in write-thread rejections. If client resets align with drops, review firewall/idle-timeout behavior and keep-alive settings. Enabling a dead-letter or persistent queue at the Gateway would help quantify and retain failed events.

How is this working? It is not clear how the data is being indexed here, what tools are involved?

Also, which tool generated this log?

"Event drop for destination.elasticsearch due to slow/unreachable destination."

You need to provide more context about these logs are being indexed and what tools are involved.

Welcome to the community forum @Siddhi_Jain

First of all, of course you can ask for help, it's essentially what the forum is for, and people like me will try to do so. But please understand this is a no-SLA, no-response-guaranteed community forum. It's not a formal support channel. IF your elasticsearch instance(s) are covered by any sort of license, I encourage you to open a support ticket!

and

So, that likely makes sense to you, but since people like me don't know the context, it makes less sense to me. You have 2 elasticsearch clusters? A single cluster is forwarding data to itself? API Gateway is feeding one to feed another? Is there anything else in the flow, eg. Logstash or Kafka or ... ?

Maybe a block diagram might help here.


This is architecture diagram of ELK and in right side Arcchitecture shows the API gateway architecture for ELK.

Not using logstash and kafka or anything only using elasticsearch

In the API Gateway tool, we are using only Elasticsearch as the destination for forwarding logs. We have configured the ELK Elasticsearch master node IP address and the index name. The same index is subsequently created in the destination Elasticsearch cluster.

Additionally, we are running an automated script on the ELK side to change the index naming format. After this process, the data is successfully visible in Kibana.

However, we are observing a significant data count mismatch between the API Gateway and ELK.

Kindly review the configuration and assist us in identifying the root cause and resolving the issue.

Thanks for sharing the block diagram, it does help!

There's literally Logstash boxes in the block diagrams you shared, in between the Filebeat and Elasticsearch cluster on the left diagram. That one is working OK, right, no lost docs. That's quite a big clue ... :thinking:

( btw the right hand side diagram looks the same as the left hand side diagram to me, except kibana is repositioned and beats instead Filebeat ? )

I'm not your architect, and I dont really understand what your "SAG" box represents, but I'll speculate the "SAG" is trying to write bursty traffic to Elasticsearch directly, and some docs are getting lost via the "back pressure" you noted. If that data were first sent to Logstash instead, or Kafka, you might have better luck.

That's not to say you can't make it work without architectural changes, but for that you'd need to share a lot more info about how the "SAG" is specifically configured, what it is running, etc.

In passing, 2 master nodes in an elasticsearch cluster is not a clever setup, so I hope that's just a simplification in the diagram and there are at least 3. In fact 3 is often the best number.

Where exactly in your diagram is the problem? You have a couple of Logstash in it, you need to be more clear where the issue you are facing is happening, also Elasticsearch does not send data by itself, it only receives data, you mentioned that the API Gateway data is being forwarded from Elasticsearch to Elasticsearch, this is not possible with Elasticsearch alone as it does not send data.

This would be possible if you were using Cross Cluster Replication between 2 clusters, in this way Elasticsearch would be able to sync the data between cluster, but this is a paid license and you would need a license to each cluster, since you didn't mention Cross Cluster Replication I'm assuming you are not using it.

So, you need to make it clear what you mean with elasticsearch-to-elasticsearch, also you didn't explain which tool generated the log you shared before.

For the community to be able to provide any help or suggestions, we need to understand exactly how you are indexing this data and it is not clear yet.