Data lost in Elasticsearch index

Elastic Stack Elasticsearch
1

Hi,
I'm using elasticsearch-jdbc importer, which inserts data from MS SQL to Elasticsearch indexes.
Today I've faced an issue: data losts in one of the indexes.

Neither elasticsearch-jdbc not elasticsearch.log contains any error messages.

A part of the elasticsearch-jdbc importer is below:

 [13:31:00,772][DEBUG][importer.jdbc.source.standard][pool-3-thread-1]   org.xbib.elasticsearch.jdbc.strategy.standard.StandardSource@4d4773ed using read connection ConnectionID:1 ClientConnectionId: ae7ffaa4-c262-483f-bbc5-eaa2ac199ae6 for executing query
 [13:42:12,218][DEBUG][importer.jdbc.source.standard][pool-3-thread-1] merged 1893926 rows
[13:42:12,219][DEBUG][importer.jdbc.source.standard][pool-3-thread-1] shutdown
[13:42:12,219][DEBUG][importer.jdbc.source.standard][pool-3-thread-1] read connection closed
[13:42:12,219][DEBUG][importer.jdbc.source.standard][pool-3-thread-1] write connection closed

And some information about index:

before import

health status index           pri rep docs.count docs.deleted store.size pri.store.size
green  open   alltrades         5   0  113443017       553746     18.5gb         18.5gb

after import

health status index           pri rep docs.count docs.deleted store.size pri.store.size
green  open   alltrades         5   0  115336943       553746     19.3gb         19.3gb

So index had increased for 1893926 records (the same number as "MSSQL exporter" sent to Elsticsearch ) but Kibana/Grafana couldn't find these data.

Can you please advise how to find these data and avoid this kind of issue?

Thanks in advance!

2

JDBC importer executes a refresh after each cycle, so the docs should be searchable.

To be extra safe, you could execute a refresh again, and check the index with curl or sense or another tool besides Kibana.