Missing indexed documents

Elastic Stack Elasticsearch
1

Hi,

I am using elasticsearch 0.20.2 on 8GB RAM system (EC2 with Arch Linux). I notice something weird happen. One day i used regular search to find some documents and i got them, but two days later those documents disappeared. I try everything to find them but they just missing. And my code (using pycassa) do not using deleting.

let me explain you my case:

  1. i have three nodes (three EC2 instances) with 5 shareds and 1 replica.
  2. i have index with >9 000 000 indexed documents.
  3. each document is lower than 20KB.
  4. new documents are constantly added into index (with 60 sec pause on each 10240 inserted documents).

First i was thinking that id is not unique but this is not cause - checked.
After that i was suspicious on limits for file descriptors and i set them to 65000 (soft and hard). elasticsearch is started with these limitations - checked. But when i do lsof -a -p <es_pid> | wc -l i see only 2090 open files and also i see that only opened are those /0/indices/index/* but why other numbers are not opened (e.g. /1/indices/index/*).

Here is cluster/index health state:

{"cluster_name":"index_cluster","status":"yellow","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":5,"active_shards":11,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":7}

This is very dangerous i really don't know how documents could disappeared.
Can someone help be with this.

Thank you,
Milan