Messages getting lost while insertion

Elastic Stack Elasticsearch
1

Hi All,
I am writing messages into elasticsearch using a single thread, and using
the call:
transportClient.prepareIndex(index, type, id)
.setSource(node)
.execute()
.actionGet();
If I write messages continuously, I observe that some messages are lost
somewhere in transit, and my query returns less entries than I write.

If I use elasticsearchhead UI, I see the status of the index as
test
size: 68.9mb (68.9mb)
docs: 42795 (42820)

My index status returns:
docs: {

  • num_docs: 42795
  • max_doc: 42820
  • deleted_docs: 25

}

The max_docs is less than the number of entries that I pushed into
ElasticSearch. And why are some messages marked as deleted?
I am using ElasticSearch 0.16.2.

Regards,
Sameer

2

Sameer,

Are you updating documents that already exist in the index? When you
update a document (insert a document with an existing id), then the
old document gets deleted and the new one inserted. The deleted
document will disappear once a merge occurs.

Ivan

On Tue, Sep 13, 2011 at 5:18 AM, Sameer Pokarna sameerpokarna@gmail.com wrote:

Hi All,
I am writing messages into elasticsearch using a single thread, and using
the call:
transportClient.prepareIndex(index, type, id)
.setSource(node)
.execute()
.actionGet();
If I write messages continuously, I observe that some messages are lost
somewhere in transit, and my query returns less entries than I write.
If I use elasticsearchhead UI, I see the status of the index as
test
size: 68.9mb (68.9mb)
docs: 42795 (42820)
My index status returns:
docs: {

num_docs: 42795
max_doc: 42820
deleted_docs: 25

}
The max_docs is less than the number of entries that I pushed into
Elasticsearch. And why are some messages marked as deleted?
I am using Elasticsearch 0.16.2.

Regards,
Sameer