Dear Team,
We are currently experiencing a significant data count mismatch between the API Gateway and ELK. We would like to highlight the issue, along with the observations and validations performed so far.
Issue Background:
Initially, when we cross-verified the data count between the API Gateway and ELK, the counts were matching correctly. At that time, only 12 APIs were live.
Subsequently, additional APIs were onboarded into ELK. However, the data count was not cross-verified immediately after onboarding. During our recent validation, we observed a significant mismatch of approximately 90% between the API Gateway and ELK data counts.
Observations and Validations Performed:
Elasticsearch Logs: We reviewed the Elasticsearch logs to identify any errors or failures related to data ingestion. However, no relevant errors were identified during the investigation.
Elasticsearch Thread Pool: We checked the Elasticsearch write thread pool statistics to identify any rejected requests. However, no rejected requests were observed during the validation.
API Gateway Logs: While reviewing the Gateway logs, we identified the following error:
"Event drop for destination.elasticsearch due to slow/unreachable destination."
This indicates that some events are being dropped at the Gateway end while forwarding data to the destination Elasticsearch.
Firewall Validation: We also coordinated with the firewall team to verify whether any packets were being dropped during communication between the source and destination. During the validation, client resets were observed.
Additional Application Validation: As part of further investigation, we cross-verified the data count for another application whose logs are being ingested through Filebeat. The data count for this application is matching correctly. However, the API Gateway data, which is being forwarded from Elasticsearch to Elasticsearch, continues to show a significant mismatch.
Request for Further Investigation:
Based on the above observations, we request your assistance in identifying the root cause of the issue and providing an appropriate resolution.
Kindly help us investigate the following:
The reason behind the event drops at the API Gateway end.
The cause of the slow/unreachable destination error and client resets.
Whether there are any limitations or bottlenecks in the Elasticsearch-to-Elasticsearch data forwarding mechanism, particularly after the onboarding of additional APIs.
Any configuration changes or optimizations required to ensure that all events are successfully delivered to the destination Elasticsearch without data loss.
Your assistance in identifying the root cause and resolving this issue would be highly appreciated, as the data count mismatch is impacting the accuracy of monitoring and reporting in ELK.
Looking forward to your response and support
