Sure thing. Here are my mappings and index settings.
parent
{
"index": {
"number_of_shards": 10,
"number_of_replicas": 0
}
}
'{
"user_meta": {
"type": "object",
"index_analyzer": "keyword",
"search_analyzer": "keyword",
"properties": {
"ks_uid": {"type": "string"},
"networks": {"type": "integer", "index_name": "network"},
"registered_flag" : {"type" : "integer"},
"tw_follower_cnt": {"type" : "long"},
"ks_score" : {"type" : "integer"},
"gender" : {"type" : "string" },
"age": {"type": "integer"},
"city" : {"type" : "string"},
"state" : {"type" : "string"},
"country" : {"type" : "string"},
"location" : {"type" : "string", "index" : "not_analyzed"},
"zip" : {"type": "string"},
"is_brand" : {"type": "integer"},
"maritalstatus" : {"type": "string"},
"phone" : {"type": "string"},
"topics" : {
"type" : "nested",
"properties": {
"topic_id": {"type": "string"},
"topic_score" : {"type": "double"}
}
},
"network_contributions" : {
"type" : "nested",
"properties": {
"network": {"type": "string"},
"raw_score" : {"type": "double"}
}
},
"email" : {"type": "string"}
}
}
}'
children
'{
"engagement": {
"type": "object",
"index_analyzer": "keyword",
"search_analyzer": "keyword",
"_parent": {"type": "user_meta"},
"properties": {
"klout_id": {"type": "string"},
"3days_active": {"type": "boolean"},
"7days_active": {"type": "boolean"},
"14days_active": {"type": "boolean"},
"30days_active": {"type": "boolean"},
"3days_active_hours": {"type": "integer"},
"7days_active_hours": {"type": "integer"},
"14days_active_hours": {"type": "integer"},
"30days_active_hours": {"type": "integer"}
}
}
}'
'{
"experts": {
"type": "object",
"index_analyzer": "keyword",
"search_analyzer": "keyword",
"_parent": {"type": "user_meta"},
"properties": {
"klout_id": {"type": "string"},
"10days_active_experts_flag": {"type": "boolean"},
"30days_active_experts_flag": {"type": "boolean"},
"90days_active_experts_flag": {"type": "boolean"},
"alltime_active_experts_flag": {"type": "boolean"},
"10days_experts_answers_count": {"type": "integer"},
"30days_experts_answers_count": {"type": "integer"},
"90days_experts_answers_count": {"type": "integer"},
"alltime_experts_answers_count": {"type": "integer"}
}
}
}'
There are three experiments I've tried:
- Index 4M parent documents. Run optimize (no flush or refresh
beforehand.) It appears that I lose documents.
- Index 4M parent documents. Index 800K children documents. It appears
that I lose documents.
- Index 4M parent documents. Add replica to the index. Index 800K children
documents. Everything looks fine at first, but overnight I lose documents.
Let me know if there are more details you would like. Unfortunately I can't
give you the documents as it includes confidential user information.
Thanks!
Jieren
On Thursday, May 30, 2013 12:09:54 AM UTC-7, Alexander Reelsen wrote:
Hey,
can you try to break down your issue maybe, so we can reproduce it
locally? Pretty hard to tell anything from the distance. Are you doing a
flush/refresh before running optimize, so you know you get back the latest
documents only before you start counting?
--Alex
On Thu, May 30, 2013 at 7:20 AM, David Pilato <da...@pilato.fr<javascript:>
wrote:
Can you see anything in node logs or in bulk requests response?
--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 30 mai 2013 à 00:50, jieren <jie...@klout.com <javascript:>> a écrit :
Hey everyone
Seeing a weird issue on 0.90.0 and I'm not sure if it's data corruption
or a bug. I'm kind of stumped here and am looking for new things to probe
for a solution.
I start by indexing 4,383,177 documents of index type A. I then run a
match_all query and get the correct number of documents. After things, I've
tried two things.
-
Run optimize on the index.
This causes me to consistently lose at least 1 document.
-
Run a second indexing job.
This job pushes 81,422 documents of index type B and 719,867 documents
of index type C. Both of these use index type A as a parent.
After running, I start losing documents on index type A, about one
every 30 minutes or so. I don't have a TTL set on these documents.
This is very confusing. Thoughts?
Thanks,
Jieren
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.