My configuration it's 2 master nodes + 6 data nodes. Master nodes have
128GB RAM + 2 CPU/8Core, data nodes 32GB RAM + 2CPU/6Core. I'm testing
migrate data from Oracle to ES. It is about 12TB. Now I have been migrated
~4TB (> 20mld. records, > 1200 indices, > 15000 shards). Queries are
working OK, but after indeksing about 3 TB ES cluster problems ocured
during indexing new data. Data to indexing are prepared like txt file in
form:
{ "index" : { } }
{
"gcr":"val1","a_m":"val1","b_m":"2","x_m":"233","tor_id":"3232","start":"2014-01-17
07:31:44","on":"474","ci":"003","im":"22","ei":"01172037","service":"126","class":"22","sc":"","rc":"498","for":"4"
}
{ "index" : { } }
....
and is send to cluster by curl (curl --silent --show-error --request POST
--data-binary @"$file_name" $host:9200/$idx_name/schema/_bulk)
Each file have 200000 records (after test best value).
Problem appear when I run 2 and more indeksing stream. Some operation has
been finished with timeout error, some didn't indexing all records.
And third problem is that after restart, cluster need couple hours to
allocate shards and setting status to green
I don't know have knowledge what I have to change to solve this problems.
My configuration it's 2 master nodes + 6 data nodes. Master nodes have 128GB RAM + 2 CPU/8Core, data nodes 32GB RAM + 2CPU/6Core. I'm testing migrate data from Oracle to ES. It is about 12TB. Now I have been migrated ~4TB (> 20mld. records, > 1200 indices, > 15000 shards). Queries are working OK, but after indeksing about 3 TB ES cluster problems ocured during indexing new data. Data to indexing are prepared like txt file in form:
{ "index" : { } }
{ "gcr":"val1","a_m":"val1","b_m":"2","x_m":"233","tor_id":"3232","start":"2014-01-17 07:31:44","on":"474","ci":"003","im":"22","ei":"01172037","service":"126","class":"22","sc":"","rc":"498","for":"4" }
{ "index" : { } }
....
and is send to cluster by curl (curl --silent --show-error --request POST --data-binary @"$file_name" $host:9200/$idx_name/schema/_bulk)
Each file have 200000 records (after test best value).
Problem appear when I run 2 and more indeksing stream. Some operation has been finished with timeout error, some didn't indexing all records.
And third problem is that after restart, cluster need couple hours to allocate shards and setting status to green
I don't know have knowledge what I have to change to solve this problems.
Before I started migration I checked many records counter - 1000, 2000, 5000, 100 000, etc. Im my configuration 200 000 is optimal value. Problem with timeout is not directly conected with quantity of records because now I stoped all load (indexing) processes and I can't update mapping for new indexes. Cluster now is during shards allocation process after restart. I'm trying create new one and update appropriate mapping for it. I do it by pure curl request and by perl Search::Elasticsearch with all option for cxn (NetCurl, LWP, etc.) but cluster still answare that after 30s with exception that update mapping isn't possible. I think that this problem is conected with some parameters for buffers, queues (?), but I don't know what.
When I come today morning cluster finished shards allocation (status
green). I run next load (indexing) and everything finished correctly. After
that I stoped all cluster nodes and run them again. Nodes started shards
allocation. In that time all thread pools were free:
o1 10.13.186.100 0 0 0 0 0 0 0 0 0
o2 10.13.186.110 0 0 0 0 0 0 0 0 0
h1 10.14.180.41 0 0 0 0 0 0 0 0 0
h2 10.14.180.42 0 0 0 0 0 0 0 0 0
h3 10.14.180.43 0 0 0 0 0 0 0 0 0
h4 10.14.180.44 0 0 0 0 0 0 0 0 0
h5 10.14.180.45 0 0 0 0 0 0 0 0 0
h6 10.14.180.46 0 0 0 0 0 0 0 0 0
In next step I created indexs - finished OK. After that I started update
mapping but I have error:
perl Search::Elasticsearch
[Timeout] ** [http://h1:9200]-[509] 28: Timeout was reached, called from
sub Search::Elasticsearch::Transport::try {...} at
/usr/local/share/perl5/Try/Tiny.pm line 81. With vars: {'request' =>
{'body' => {'schema' => {'properties' => .... 'qs' => {},'method' =>
'PUT','serialize' => 'std','path' =>
'/new_index_name/_mapping/schema','ignore' => ,'mime_type' =>
'application/json'},'status_code' => 509}
curl
{"error":"RemoteTransportException[[o2][inet[/10.13.186.110:9300]][indices:admin/mapping/put]];
nested: ProcessClusterEventTimeoutException[failed to process cluster event
(put-mapping [schema]) within 30s]; ","status":503}
During shards allocation some cluster resources are full utilized, but I
don't have idea witch one. It's interesing that I can create new index but
I can't update mapping.
Any suggestion?
Regards
W dniu środa, 14 stycznia 2015 20:06:31 UTC+1 użytkownik Marek Dabrowski
napisał:
David
Before I started migration I checked many records counter - 1000, 2000,
5000, 100 000, etc. Im my configuration 200 000 is optimal value. Problem
with timeout is not directly conected with quantity of records because now
I stoped all load (indexing) processes and I can't update mapping for new
indexes. Cluster now is during shards allocation process after restart. I'm
trying create new one and update appropriate mapping for it. I do it by
pure curl request and by perl Search::Elasticsearch with all option for cxn
(NetCurl, LWP, etc.) but cluster still answare that after 30s with
exception that update mapping isn't possible. I think that this problem is
conected with some parameters for buffers, queues (?), but I don't know
what.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.