Elasticsearch Troubleshooting - Missing Documents

[jgeek]

We are running elasticsearch 8.12.2 through docker-compose. We are seeing a weird issue where some documents are missing in elasticsearch. One of the third party java spring boot application (zeebe) exports its records to elasticsearch. While all of its records are pushed to elasticsearch through the elasticsearch java client, only some records are seen when queried through the elasticsearch REST api. The REST api queries all indexes and searches through all indexes. Some records are indeed seen in the response but some are missing.

We debugged the java application and confirmed that there are no filters applied and we did see that the code does call client.index() method for all records. The records are for some reason missing in elasticsearch.

I enabled TRACE log level for elasticsearch container but that didn't gave any insights.

Any hints/tips for troubleshooting this further to see what's going wrong?

REST API to query data

curl --location --request GET 'http://localhost:9200/zeebe-record_process-instance_*,zeebe-record_job_*,zeebe-record_process-event_*,zeebe-record_process_*,zeebe-record_process-instance-creation_*,zeebe-record_job-batch_*,zeebe-record_deployment_*,zeebe-record_process_*,/_search' \
--header 'Content-Type: application/json' \
--data '{
  "query": {
    "term": {
      "value.processInstanceKey": "2251799813685249"
    }
  }
}'

Thanks.

[Christian_Dahlqvist]

Does the code check the responses coming back from Elasticsearch and log any failures? One thing that can cause documents to fail indexing is mapping conflicts, and Elasticsearch would indicate this in the request response. Nothing would go into the Elasticsearch logs though.

If you have a document that failed to get indexed you can always try to index it manually using e.g. curl or the Kibana dev console, and see if it fails.