Losing documents on optimize or possibly refresh

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption or
a bug. I'm kind of stumped here and am looking for new things to probe for
a solution.

I start by indexing 4,383,177 documents of index type A. I then run a
match_all query and get the correct number of documents. After things, I've
tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents of
    index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one every
    30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Can you see anything in node logs or in bulk requests response?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 00:50, jieren jieren@klout.com a écrit :

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption or a bug. I'm kind of stumped here and am looking for new things to probe for a solution.

I start by indexing 4,383,177 documents of index type A. I then run a match_all query and get the correct number of documents. After things, I've tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents of index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one every 30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

can you try to break down your issue maybe, so we can reproduce it locally?
Pretty hard to tell anything from the distance. Are you doing a
flush/refresh before running optimize, so you know you get back the latest
documents only before you start counting?

--Alex

On Thu, May 30, 2013 at 7:20 AM, David Pilato david@pilato.fr wrote:

Can you see anything in node logs or in bulk requests response?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 00:50, jieren jieren@klout.com a écrit :

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption or
a bug. I'm kind of stumped here and am looking for new things to probe for
a solution.

I start by indexing 4,383,177 documents of index type A. I then run a
match_all query and get the correct number of documents. After things, I've
tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents
    of index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one every
    30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Nope. I only have the slow log set to INFO. Are there additional logging
settings I should enable?

Thanks

On Wednesday, May 29, 2013 10:20:45 PM UTC-7, David Pilato wrote:

Can you see anything in node logs or in bulk requests response?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 00:50, jieren <jie...@klout.com <javascript:>> a écrit :

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption or
a bug. I'm kind of stumped here and am looking for new things to probe for
a solution.

I start by indexing 4,383,177 documents of index type A. I then run a
match_all query and get the correct number of documents. After things, I've
tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents
    of index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one every
    30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

How do you count your number of docs?
Do you use _count or _search API?

So you only index docs, don't you?

Can you ask for index status? Just after injection on a new index and after 30 minutes?
http://www.elasticsearch.org/guide/reference/api/admin-indices-status/

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 19:57, jieren jieren@klout.com a écrit :

Nope. I only have the slow log set to INFO. Are there additional logging settings I should enable?

Thanks

On Wednesday, May 29, 2013 10:20:45 PM UTC-7, David Pilato wrote:

Can you see anything in node logs or in bulk requests response?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 00:50, jieren jie...@klout.com a écrit :

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption or a bug. I'm kind of stumped here and am looking for new things to probe for a solution.

I start by indexing 4,383,177 documents of index type A. I then run a match_all query and get the correct number of documents. After things, I've tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents of index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one every 30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

How do you count your number of docs?
Do you use _count or _search API?
I was using the _search API, but the _count API is giving me similar
results.

So you only index docs, don't you?
Yep. I index a set of documents and then index child documents for those.

Can you ask for index status? Just after injection on a new index and
after 30 minutes?

Will do after this next index. I just checked status on the old index and
saw a bunch of deleted_docs. What could be the reasoning for this if I
don't explicitly delete anything? Could it have something to do with
indexing parent/child docs?

On Thursday, May 30, 2013 11:13:25 AM UTC-7, David Pilato wrote:

How do you count your number of docs?
Do you use _count or _search API?

So you only index docs, don't you?

Can you ask for index status? Just after injection on a new index and
after 30 minutes?
http://www.elasticsearch.org/guide/reference/api/admin-indices-status/

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 19:57, jieren <jie...@klout.com <javascript:>> a écrit :

Nope. I only have the slow log set to INFO. Are there additional logging
settings I should enable?

Thanks

On Wednesday, May 29, 2013 10:20:45 PM UTC-7, David Pilato wrote:

Can you see anything in node logs or in bulk requests response?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 00:50, jieren jie...@klout.com a écrit :

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption
or a bug. I'm kind of stumped here and am looking for new things to probe
for a solution.

I start by indexing 4,383,177 documents of index type A. I then run a
match_all query and get the correct number of documents. After things, I've
tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents
    of index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one
    every 30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sure thing. Here are my mappings and index settings.
parent
{
"index": {

    "number_of_shards": 10,                                             
                                                 
    "number_of_replicas": 0                                             
                                                 
}                                                                       

}

'{
"user_meta": {

    "type": "object",                                                   
                                                 
    "index_analyzer": "keyword",                                       
                                                  
    "search_analyzer": "keyword",                                       
                                                 
    "properties": {                                                     
                                                 
        "ks_uid": {"type": "string"},                                   
                                                 
        "networks": {"type": "integer", "index_name": "network"},       
                                                 
        "registered_flag" : {"type" : "integer"},                       
                                                 
        "tw_follower_cnt": {"type" : "long"},                           
                                                 
        "ks_score" : {"type" : "integer"},                             
                                                  
        "gender" : {"type" : "string" },                               
                                                  
            "age": {"type": "integer"},                                 
                                                 
        "city" : {"type" : "string"},                                   
                                                 
        "state" : {"type" : "string"},                                 
                                                  
        "country" : {"type" : "string"},                               
                                                  
        "location" : {"type" : "string", "index" : "not_analyzed"},     
                                                 
        "zip" : {"type": "string"},                                     
                                                 
        "is_brand" : {"type": "integer"},
        "maritalstatus" : {"type": "string"},                           
                                                 
        "phone" : {"type": "string"},                                   
                                                 
        "topics" : {                                                   
                                                  
            "type" : "nested",                                         
                                                  
            "properties": {                                             
                                                 
                "topic_id": {"type": "string"},                         
                                                 
                "topic_score" : {"type": "double"}                     
                                                  
             }                                                         
                                                  
         },                                                             
                                                 
        "network_contributions" : {                                     
                                                 
             "type" : "nested",                                         
                                                 
             "properties": {                                           
                                                  
                 "network": {"type": "string"},                         
                                                 
                 "raw_score" : {"type": "double"}                       
                                                 
             }                                                         
                                                  
        },                                                             
                                                  
        "email" : {"type": "string"}                                   
                                                  
    }                                                                   
                                                 
}                                                                       

}'

children
'{
"engagement": {

    "type": "object",                                                   
                                                 
    "index_analyzer": "keyword",                                       
                                                  
    "search_analyzer": "keyword",                                       
                                                 
    "_parent": {"type": "user_meta"},                                   
                                                 
    "properties": {                                                     
                                                 
        "klout_id": {"type": "string"},                                 
                                                 
        "3days_active": {"type": "boolean"},                           
                                                  
        "7days_active": {"type": "boolean"},                           
                                                  
        "14days_active": {"type": "boolean"},                           
                                                 
        "30days_active": {"type": "boolean"},                           
                                                 
        "3days_active_hours": {"type": "integer"},                     
                                                  
        "7days_active_hours": {"type": "integer"},                     
                                                  
        "14days_active_hours": {"type": "integer"},                     
                                                 
        "30days_active_hours": {"type": "integer"}                     
                                                  
    }                                                                   
                                                 
}                                                                       

}'

'{
"experts": {

    "type": "object",                                                   
                                                 
    "index_analyzer": "keyword",                                       
                                                  
    "search_analyzer": "keyword",                                       
                                                 
    "_parent": {"type": "user_meta"},                                   
                                                 
    "properties": {                                                     
                                                 
        "klout_id": {"type": "string"},                                 
                                                 
        "10days_active_experts_flag": {"type": "boolean"},             
                                                  
        "30days_active_experts_flag": {"type": "boolean"},             
                                                  
        "90days_active_experts_flag": {"type": "boolean"},             
                                                  
        "alltime_active_experts_flag": {"type": "boolean"},             
                                                 
        "10days_experts_answers_count": {"type": "integer"},           
                                                  
        "30days_experts_answers_count": {"type": "integer"},           
                                                  
        "90days_experts_answers_count": {"type": "integer"},           
                                                  
        "alltime_experts_answers_count": {"type": "integer"}           
                                                  
    }                                                                   
                                                 
}                                                                       

}'

There are three experiments I've tried:

  1. Index 4M parent documents. Run optimize (no flush or refresh
    beforehand.) It appears that I lose documents.
  2. Index 4M parent documents. Index 800K children documents. It appears
    that I lose documents.
  3. Index 4M parent documents. Add replica to the index. Index 800K children
    documents. Everything looks fine at first, but overnight I lose documents.

Let me know if there are more details you would like. Unfortunately I can't
give you the documents as it includes confidential user information.

Thanks!
Jieren

On Thursday, May 30, 2013 12:09:54 AM UTC-7, Alexander Reelsen wrote:

Hey,

can you try to break down your issue maybe, so we can reproduce it
locally? Pretty hard to tell anything from the distance. Are you doing a
flush/refresh before running optimize, so you know you get back the latest
documents only before you start counting?

--Alex

On Thu, May 30, 2013 at 7:20 AM, David Pilato <da...@pilato.fr<javascript:>

wrote:

Can you see anything in node logs or in bulk requests response?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 mai 2013 à 00:50, jieren <jie...@klout.com <javascript:>> a écrit :

Hey everyone

Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption
or a bug. I'm kind of stumped here and am looking for new things to probe
for a solution.

I start by indexing 4,383,177 documents of index type A. I then run a
match_all query and get the correct number of documents. After things, I've
tried two things.

  1. Run optimize on the index.
    This causes me to consistently lose at least 1 document.

  2. Run a second indexing job.
    This job pushes 81,422 documents of index type B and 719,867 documents
    of index type C. Both of these use index type A as a parent.
    After running, I start losing documents on index type A, about one
    every 30 minutes or so. I don't have a TTL set on these documents.

This is very confusing. Thoughts?

Thanks,
Jieren

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.