Timeout

Hello

My configuration it's 2 master nodes + 6 data nodes. Master nodes have
128GB RAM + 2 CPU/8Core, data nodes 32GB RAM + 2CPU/6Core. I'm testing
migrate data from Oracle to ES. It is about 12TB. Now I have been migrated
~4TB (> 20mld. records, > 1200 indices, > 15000 shards). Queries are
working OK, but after indeksing about 3 TB ES cluster problems ocured
during indexing new data. Data to indexing are prepared like txt file in
form:
{ "index" : { } }
{
"gcr":"val1","a_m":"val1","b_m":"2","x_m":"233","tor_id":"3232","start":"2014-01-17
07:31:44","on":"474","ci":"003","im":"22","ei":"01172037","service":"126","class":"22","sc":"","rc":"498","for":"4"
}
{ "index" : { } }
....
and is send to cluster by curl (curl --silent --show-error --request POST
--data-binary @"$file_name" $host:9200/$idx_name/schema/_bulk)
Each file have 200000 records (after test best value).
Problem appear when I run 2 and more indeksing stream. Some operation has
been finished with timeout error, some didn't indexing all records.
And third problem is that after restart, cluster need couple hours to
allocate shards and setting status to green
I don't know have knowledge what I have to change to solve this problems.

Any body have suggestion?

Regards
Marek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/59354a68-1ff6-46ac-8d77-01328b353f1b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I think you should send less requests within the same bulk.

You could try to send 10000 requests and do that 20 times.
I bet you’ll get better results.

Not sure that will fix your issue though.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 14 janv. 2015 à 14:32, Marek Dabrowski marek.dabrowski@gmail.com a écrit :

Hello

My configuration it's 2 master nodes + 6 data nodes. Master nodes have 128GB RAM + 2 CPU/8Core, data nodes 32GB RAM + 2CPU/6Core. I'm testing migrate data from Oracle to ES. It is about 12TB. Now I have been migrated ~4TB (> 20mld. records, > 1200 indices, > 15000 shards). Queries are working OK, but after indeksing about 3 TB ES cluster problems ocured during indexing new data. Data to indexing are prepared like txt file in form:
{ "index" : { } }
{ "gcr":"val1","a_m":"val1","b_m":"2","x_m":"233","tor_id":"3232","start":"2014-01-17 07:31:44","on":"474","ci":"003","im":"22","ei":"01172037","service":"126","class":"22","sc":"","rc":"498","for":"4" }
{ "index" : { } }
....
and is send to cluster by curl (curl --silent --show-error --request POST --data-binary @"$file_name" $host:9200/$idx_name/schema/_bulk)
Each file have 200000 records (after test best value).
Problem appear when I run 2 and more indeksing stream. Some operation has been finished with timeout error, some didn't indexing all records.
And third problem is that after restart, cluster need couple hours to allocate shards and setting status to green
I don't know have knowledge what I have to change to solve this problems.

Any body have suggestion?

Regards
Marek

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/59354a68-1ff6-46ac-8d77-01328b353f1b%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/59354a68-1ff6-46ac-8d77-01328b353f1b%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/96C066A0-BA5E-4BF5-A292-1E9D9709877E%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

David

Before I started migration I checked many records counter - 1000, 2000, 5000, 100 000, etc. Im my configuration 200 000 is optimal value. Problem with timeout is not directly conected with quantity of records because now I stoped all load (indexing) processes and I can't update mapping for new indexes. Cluster now is during shards allocation process after restart. I'm trying create new one and update appropriate mapping for it. I do it by pure curl request and by perl Search::Elasticsearch with all option for cxn (NetCurl, LWP, etc.) but cluster still answare that after 30s with exception that update mapping isn't possible. I think that this problem is conected with some parameters for buffers, queues (?), but I don't know what.

Regards

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dbaa9cd3-65fa-4117-8c3b-bfd6096611f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I'm doing next test...

When I come today morning cluster finished shards allocation (status
green). I run next load (indexing) and everything finished correctly. After
that I stoped all cluster nodes and run them again. Nodes started shards
allocation. In that time all thread pools were free:
o1 10.13.186.100 0 0 0 0 0 0 0 0 0
o2 10.13.186.110 0 0 0 0 0 0 0 0 0
h1 10.14.180.41 0 0 0 0 0 0 0 0 0
h2 10.14.180.42 0 0 0 0 0 0 0 0 0
h3 10.14.180.43 0 0 0 0 0 0 0 0 0
h4 10.14.180.44 0 0 0 0 0 0 0 0 0
h5 10.14.180.45 0 0 0 0 0 0 0 0 0
h6 10.14.180.46 0 0 0 0 0 0 0 0 0
In next step I created indexs - finished OK. After that I started update
mapping but I have error:

  1. perl Search::Elasticsearch
    [Timeout] ** [http://h1:9200]-[509] 28: Timeout was reached, called from
    sub Search::Elasticsearch::Transport::try {...} at
    /usr/local/share/perl5/Try/Tiny.pm line 81. With vars: {'request' =>
    {'body' => {'schema' => {'properties' => .... 'qs' => {},'method' =>
    'PUT','serialize' => 'std','path' =>
    '/new_index_name/_mapping/schema','ignore' => ,'mime_type' =>
    'application/json'},'status_code' => 509}

  2. curl
    {"error":"RemoteTransportException[[o2][inet[/10.13.186.110:9300]][indices:admin/mapping/put]];
    nested: ProcessClusterEventTimeoutException[failed to process cluster event
    (put-mapping [schema]) within 30s]; ","status":503}

During shards allocation some cluster resources are full utilized, but I
don't have idea witch one. It's interesing that I can create new index but
I can't update mapping.

Any suggestion?

Regards

W dniu środa, 14 stycznia 2015 20:06:31 UTC+1 użytkownik Marek Dabrowski
napisał:

David

Before I started migration I checked many records counter - 1000, 2000,
5000, 100 000, etc. Im my configuration 200 000 is optimal value. Problem
with timeout is not directly conected with quantity of records because now
I stoped all load (indexing) processes and I can't update mapping for new
indexes. Cluster now is during shards allocation process after restart. I'm
trying create new one and update appropriate mapping for it. I do it by
pure curl request and by perl Search::Elasticsearch with all option for cxn
(NetCurl, LWP, etc.) but cluster still answare that after 30s with
exception that update mapping isn't possible. I think that this problem is
conected with some parameters for buffers, queues (?), but I don't know
what.

Regards

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ab6d96f-e5cc-4ce1-b528-acedccef7ccf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.