How to track why total number of docs is decreasing?

Elastic Stack Elasticsearch
1

I have lots of documents being indexed into an ES cluster, and I do not expect we are making use of TTLs in any index/mapping, nor do i expect we have any process that should ever delete documents... but the total number of documents, as reported by kopf plugin, is decreasing on regular basis...

How can I track down why is it this happening? Is there some query logging maybe to enable? or could this number be decreasing for reasons other than "someone sent a delete document" request?

Thanks a lot,
Mohamed.

2

Does the issue replicate if you get a count of documents using the Elasticsearch APIs? If not, the issue is with the kopf plugin. If so, can you let us know and we'll probably need to gather more information from you? Let's a get a replication first solely within Elasticsearch though.

3

Thanks. Like so:
`[root@rap-es2 ~]# for i in $(seq 10);do !!;sleep 10;done
for i in $(seq 10);do curl 'http://localhost:9200/_cat/count?v';echo;sleep 10;done
epoch timestamp count
1452830455 23:00:55 621610925

epoch timestamp count
1452830465 23:01:05 621610925

epoch timestamp count
1452830475 23:01:15 621610925

epoch timestamp count
1452830485 23:01:25 621610854

epoch timestamp count
1452830496 23:01:36 621610821

^C
[root@rap-es2 ~]#
`

4

Never used it but about checking the queries sent via slow.log? https://www.elastic.co/guide/en/elasticsearch/reference/1.4/index-modules-slowlog.html

5

Can you please verify that this is in fact the case? You should be able to do this with the get mappings API:

$ curl -XGET localhost:9200/_all/_mapping | grep _ttl

or the equivalent of your choice.

6

maybe I have a too old ES?
➜ ES rpm -qa| grep elas elasticsearch-1.2.4-1.noarch ➜ ES curl -s -XGET localhost:9200/_all/_mappings ;echo No handler found for uri [/_all/_mappings] and method [GET] ➜ ES

7

I removed the s from mappings... Thanks!

and thank you so much!! I do have _ttl in all my indexes!!!! I can believe all the docs I lost.. Grrrrr!!!

"en_2015": { "mappings": { "doc": { "_source": { "enabled": false }, "_ttl": { "default": 7776000000, "enabled": true }, ...

8

That makes a lot more sense. I'm glad that we were able to track it down. :slight_smile:

9

One more question Jason.... How do I fix this now? This is defined in many (actually not all) of my indexes.... I guess I need to update the mapping for each... but would that also remove the ttl for already indexed documents?

10

Correct, you need to update the mappings. Disabling the TTL on the index level will stop the purge thread from purging expired documents from the index, but it will not remove the _ttl field from the documents.

11

Thanks... and disabling the purge is not available in 1.2 right? it is this setting: index.ttl.disable_purge ?
I see it in 1.7 docs as experimental: https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-update-settings.html

12

I think that's been there since #1791 which is pre-1.2. But it's exactly because that flag is marked as experimental that I would update your mappings.