# Missing data sent to be indexed **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2010, 11:08pm UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121 "2010-07-19T23:08:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![David\_Jensen\_2](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@David\_Jensen\_2](https://discuss.elastic.co/u/David_Jensen_2)\ **Post date:** [July 19, 2010, 11:08pm UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/1 "2010-07-19T23:08:37Z")
I've tried this a few times now. I send 15 documents to be indexed. When I check status on the servers, the num\_docs is only incremented by 5. I'm running on 2 large EC2 instances with an S3 gateway. I have 10 shards and 2 replicas. Is there some weirdness because I'm configured for 2 replicas but I only have enough machines for 1 replica? ---
**Author:** ![David\_Jensen\_2](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@David\_Jensen\_2](https://discuss.elastic.co/u/David_Jensen_2)\ **Post date:** [July 19, 2010, 11:19pm UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/2 "2010-07-19T23:19:56Z")
I've just now verified that adding another instance did \>not\< solve the problem. So the question is, why isn't ES indexing my data OR what the heck am I doing wrong? On Jul 19, 4:08 pm, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > I've tried this a few times now. I send 15 documents to be indexed. > > When I check status on the servers, the num\_docs is only incremented > by 5. > > I'm running on 2 large EC2 instances with an S3 gateway. I have 10 > shards and 2 replicas. > > Is there some weirdness because I'm configured for 2 replicas but I > only have enough machines for 1 replica? ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [July 19, 2010, 11:24pm UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/3 "2010-07-19T23:24:18Z")
I take it you have no error messages in the logs ? May be you can increase the log level to observe what it says when it says the data. If you can post the data that does not get indexed as well as the index mapping, it may have some clues to what the problem may be. What version of ES are you using? Just to cover the basics, you're checking the number of docs after waiting couple of secs right? Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Mon, Jul 19, 2010 at 7:19 PM, David Jensen [djensen47@gmail.com](mailto:djensen47@gmail.com) wrote: > I've just now verified that adding another instance did \>not\< solve > the problem. So the question is, why isn't ES indexing my data OR what > the heck am I doing wrong? > > On Jul 19, 4:08 pm, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > > > I've tried this a few times now. I send 15 documents to be indexed. > > > > When I check status on the servers, the num\_docs is only incremented > > by 5. > > > > I'm running on 2 large EC2 instances with an S3 gateway. I have 10 > > shards and 2 replicas. > > > > Is there some weirdness because I'm configured for 2 replicas but I > > only have enough machines for 1 replica? ---
**Author:** ![David\_Jensen\_2](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@David\_Jensen\_2](https://discuss.elastic.co/u/David_Jensen_2)\ **Post date:** [July 19, 2010, 11:53pm UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/4 "2010-07-19T23:53:27Z")
There is another thread with the exception; has a title like "75k missing docs". After clearing out the index completely and loading a new clean record, everything seems to be happy. On Jul 19, 4:24 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > I take it you have no error messages in the logs ? May be you can increase > the log level to observe what it says when it says the data. > If you can post the data that does not get indexed as well as the index > mapping, it may have some clues to what the problem may be. What version of > ES are you using? > Just to cover the basics, you're checking the number of docs after waiting > couple of secs right? > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Mon, Jul 19, 2010 at 7:19 PM, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > > > I've just now verified that adding another instance did \>not\< solve > > the problem. So the question is, why isn't ES indexing my data OR what > > the heck am I doing wrong? > > > On Jul 19, 4:08 pm, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > > > > > I've tried this a few times now. I send 15 documents to be indexed. > > > > When I check status on the servers, the num\_docs is only incremented > > > by 5. > > > > I'm running on 2 large EC2 instances with an S3 gateway. I have 10 > > > shards and 2 replicas. > > > > Is there some weirdness because I'm configured for 2 replicas but I > > > only have enough machines for 1 replica? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 20, 2010, 7:28am UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/5 "2010-07-20T07:28:58Z")
Is there a chance that you are simply running into the "near real time" aspect of elasticsearch? If not, can you execute a search request similar to: curl -XGET [http://host:9200/\_search?q=\*:](http://host:9200/_search?q=*:)\* and check that the total\_hits (thats basically a search all query)? -shay.banon On Tue, Jul 20, 2010 at 2:53 AM, David Jensen [djensen47@gmail.com](mailto:djensen47@gmail.com) wrote: > There is another thread with the exception; has a title like "75k > missing docs". > > After clearing out the index completely and loading a new clean > record, everything seems to be happy. > > On Jul 19, 4:24 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > > > I take it you have no error messages in the logs ? May be you can > > increase > > the log level to observe what it says when it says the data. > > If you can post the data that does not get indexed as well as the index > > mapping, it may have some clues to what the problem may be. What version > > of > > ES are you using? > > Just to cover the basics, you're checking the number of docs after > > waiting > > couple of secs right? > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Mon, Jul 19, 2010 at 7:19 PM, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) > > wrote: > > > > > I've just now verified that adding another instance did \>not\< solve > > > the problem. So the question is, why isn't ES indexing my data OR what > > > the heck am I doing wrong? > > > > > On Jul 19, 4:08 pm, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > > > > > > > I've tried this a few times now. I send 15 documents to be indexed. > > > > > > When I check status on the servers, the num\_docs is only incremented > > > > by 5. > > > > > > I'm running on 2 large EC2 instances with an S3 gateway. I have 10 > > > > shards and 2 replicas. > > > > > > Is there some weirdness because I'm configured for 2 replicas but I > > > > only have enough machines for 1 replica? ---
**Author:** ![David\_Jensen\_2](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@David\_Jensen\_2](https://discuss.elastic.co/u/David_Jensen_2)\ **Post date:** [July 20, 2010, 5:15pm UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/6 "2010-07-20T17:15:02Z")
It looked like ES was rejecting some of my records. See this thread: [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/4d74faac7dab497e#](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/4d74faac7dab497e#) I think what I need to do is define the index before loading or at least start with a known good document to index. Once I cleared the index and started from scratch with a good document, the exceptions stopped. On Jul 20, 12:28 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Is there a chance that you are simply running into the "near real time" > aspect of elasticsearch? If not, can you execute a search request similar > to: curl -XGEThttp://host:9200/\_search?q=\*:\*and check that the total\_hits > (thats basically a search all query)? > > -shay.banon > > On Tue, Jul 20, 2010 at 2:53 AM, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > > > There is another thread with the exception; has a title like "75k > > missing docs". > > > After clearing out the index completely and loading a new clean > > record, everything seems to be happy. > > > On Jul 19, 4:24 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > > > > > I take it you have no error messages in the logs ? May be you can > > > increase > > > the log level to observe what it says when it says the data. > > > If you can post the data that does not get indexed as well as the index > > > mapping, it may have some clues to what the problem may be. What version > > > of > > > ES are you using? > > > Just to cover the basics, you're checking the number of docs after > > > waiting > > > couple of secs right? > > > > Regards, > > > Berkay Mollamustafaoglu > > > mberkay on yahoo, google and skype > > > > On Mon, Jul 19, 2010 at 7:19 PM, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) > > > wrote: > > > > > > > I've just now verified that adding another instance did \>not\< solve > > > > the problem. So the question is, why isn't ES indexing my data OR what > > > > the heck am I doing wrong? > > > > > On Jul 19, 4:08 pm, David Jensen [djense...@gmail.com](mailto:djense...@gmail.com) wrote: > > > > > > > > > I've tried this a few times now. I send 15 documents to be indexed. > > > > > > When I check status on the servers, the num\_docs is only incremented > > > > > by 5. > > > > > > I'm running on 2 large EC2 instances with an S3 gateway. I have 10 > > > > > shards and 2 replicas. > > > > > > Is there some weirdness because I'm configured for 2 replicas but I > > > > > only have enough machines for 1 replica? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/missing-data-sent-to-be-indexed/3121/7 "2017-07-06T04:22:03Z")