Hi,
I am using elasticsearch 0.20.2 on 8GB RAM system (EC2 with Arch Linux). I notice something weird happen. One day i used regular search to find some documents and i got them, but two days later those documents disappeared. I try everything to find them but they just missing. And my code (using pycassa) do not using deleting.
let me explain you my case:
- i have three nodes (three EC2 instances) with 5 shareds and 1 replica.
- i have index with >9 000 000 indexed documents.
- each document is lower than 20KB.
- new documents are constantly added into index (with 60 sec pause on each 10240 inserted documents).
First i was thinking that id is not unique but this is not cause - checked.
After that i was suspicious on limits for file descriptors and i set them to 65000 (soft and hard). elasticsearch is started with these limitations - checked. But when i do lsof -a -p <es_pid> | wc -l i see only 2090 open files and also i see that only opened are those /0/indices/index/* but why other numbers are not opened (e.g. /1/indices/index/*).
Here is cluster/index health state:
{"cluster_name":"index_cluster","status":"yellow","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":5,"active_shards":11,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":7}
This is very dangerous i really don't know how documents could disappeared.
Can someone help be with this.
Thank you,
Milan