Why is ES memory consumption raising while I bulk index parent/child documents?

David_MZ · June 20, 2013, 8:38am

I am indexing parent/child documents at 1500/second rate, my index size is
57million records combined (currently as I write this) and I do not perform
any search while I index

My memory consumption is raising steady while I index, speed of 1 gig per
20 minutes
What is the reason for this, if I shutdown and restart It drops and rises
again

The indexing speed is reduced after some time

I would like to control and maintain index speed, please advice

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

radu_gheorghe · June 20, 2013, 8:48am

Hi David,

I think it's merginghttp://www.elasticsearch.org/guide/reference/index-modules/merge/that
lowers your indexing speed as the index gets bigger, not the increased
memory.

At one moment, your memory should get freed up by the Garbage Collector, so
the steady increase feels normal up to a point.

If you need to maintain the indexing speed, the solution depends on how
your data looks like. For time-rolling data, such as logs, a good thing
would be to have time-based indices (eg: one per day). This way, you have
some control over how big your indices will become.

On Thu, Jun 20, 2013 at 11:38 AM, David MZ david.mazvovsky@gmail.comwrote:

I am indexing parent/child documents at 1500/second rate, my index size is
57million records combined (currently as I write this) and I do not perform
any search while I index

My memory consumption is raising steady while I index, speed of 1 gig per
20 minutes
What is the reason for this, if I shutdown and restart It drops and rises
again

The indexing speed is reduced after some time

I would like to control and maintain index speed, please advice

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

David_MZ · June 20, 2013, 8:53am

"At one moment, your memory should get freed up by the Garbage Collector,
so the steady increase feels normal up to a point."

You mean that the increase will be capped?

The nature of my data is data about a user, all of the documents are almost
the same size, but there are 150 fields per document but I do not analyze
them as they are one word fields.
I played with merging policy, set the number of concurrent merges to 2

Are there more settings to set to improve performance

The size of the data set is 180 million from which I generate 100 million
parents

On Thu, Jun 20, 2013 at 11:48 AM, Radu Gheorghe
radu.gheorghe@sematext.comwrote:

Hi David,

I think it's merginghttp://www.elasticsearch.org/guide/reference/index-modules/merge/that lowers your indexing speed as the index gets bigger, not the increased
memory.

At one moment, your memory should get freed up by the Garbage Collector,
so the steady increase feels normal up to a point.

If you need to maintain the indexing speed, the solution depends on how
your data looks like. For time-rolling data, such as logs, a good thing
would be to have time-based indices (eg: one per day). This way, you have
some control over how big your indices will become.

On Thu, Jun 20, 2013 at 11:38 AM, David MZ david.mazvovsky@gmail.comwrote:

I am indexing parent/child documents at 1500/second rate, my index size
is 57million records combined (currently as I write this) and I do not
perform any search while I index

My memory consumption is raising steady while I index, speed of 1 gig per
20 minutes
What is the reason for this, if I shutdown and restart It drops and rises
again

The indexing speed is reduced after some time

I would like to control and maintain index speed, please advice

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/rKjqMtgSqcE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

radu_gheorghe · June 20, 2013, 10:28am

On Thu, Jun 20, 2013 at 11:53 AM, David MZ david.mazvovsky@gmail.comwrote:

"At one moment, your memory should get freed up by the Garbage Collector,
so the steady increase feels normal up to a point."

You mean that the increase will be capped?

Normally, yes. Although garbage collecting is CPU-intensive, which might
hurt performance for that time. But you should test and see how it all
works for you.

The nature of my data is data about a user, all of the documents are
almost the same size, but there are 150 fields per document but I do not
analyze them as they are one word fields.
I played with merging policy, set the number of concurrent merges to 2

Are there more settings to set to improve performance

Indexing performance? Yes, but it depends on your use-case. Have a look at:

bulk indexing
increasing or even disabling refresh_interval during indexing
translog parameters
indices.memory.index_buffer_size

Ah, here's an interesting gist:

gist.github.com

https://gist.github.com/duydo/2427158

elasticsearch_best_practices.txt

If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:

- Use create in the index API (assuming you can).
- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
- Increase the memory allocated to elasticsearch node. By default its 1g.
- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
- Increase the number of machines you have so you get less shards allocated per machine.
- Increase the number of shards an index has, so it can make use of more machines.

This file has been truncated. show original

And you should also monitor your cluster while testing to see what's your
bottleneck. This will help you know what makes sense to improve. If you
need a monitoring tool, check our SPM:

Best regards,
Radu

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · June 20, 2013, 12:09pm

Am 20.06.13 10:38, schrieb David MZ:

I am indexing parent/child documents at 1500/second rate, my index
size is 57million records combined (currently as I write this) and I
do not perform any search while I index

My memory consumption is raising steady while I index, speed of 1 gig
per 20 minutes

How much is your memory consumption? What process? How much nodes do you
have in your cluster?

What is the reason for this, if I shutdown and restart It drops and
rises again

Do you shutdown server or client? If you start a process, memory usage
will start again, nothing to worry about.

The indexing speed is reduced after some time

How much is the reduced indexing speed?

I would like to control and maintain index speed, please advice

Please give additional information about your server and client config
and the numbers you observed.

If you index millions of documents, ES will have more work to manage
growing Lucene segments and merging them, which will consume all of the
available CPU and memory, mostly observable as sudden slowdowns
("spikes"). If you throttle indexing, you can better handle the phase of
merging spikes. There are many methods you can choose from to streamline
indexing, on client or on server side.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

David_MZ · June 20, 2013, 1:14pm

"How much is your memory consumption? What process? How much nodes do you
have in your cluster? "

I have only one computer, so 1 node with 3 shards

"How much is the reduced indexing speed? "

The speed drops by 50% and later by 100%, sometimes I got spikes of very
bad performance, I guess it's GC or a merge

Question?

As I do not need to search the index while indexing, can I not do any
merges at all while indexing, and just do that at the end
my system is not real time at any sense, I am preparing a static index to
do read only

I have enough disk space (I think) to have it un merged

On Thursday, June 20, 2013 3:09:02 PM UTC+3, Jörg Prante wrote:

Am 20.06.13 10:38, schrieb David MZ:

I am indexing parent/child documents at 1500/second rate, my index
size is 57million records combined (currently as I write this) and I
do not perform any search while I index

My memory consumption is raising steady while I index, speed of 1 gig
per 20 minutes

How much is your memory consumption? What process? How much nodes do you
have in your cluster?

What is the reason for this, if I shutdown and restart It drops and
rises again

Do you shutdown server or client? If you start a process, memory usage
will start again, nothing to worry about.

The indexing speed is reduced after some time

How much is the reduced indexing speed?

I would like to control and maintain index speed, please advice

Please give additional information about your server and client config
and the numbers you observed.

If you index millions of documents, ES will have more work to manage
growing Lucene segments and merging them, which will consume all of the
available CPU and memory, mostly observable as sudden slowdowns
("spikes"). If you throttle indexing, you can better handle the phase of
merging spikes. There are many methods you can choose from to streamline
indexing, on client or on server side.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · June 20, 2013, 1:22pm

Am 20.06.13 15:14, schrieb David MZ:

The speed drops by 50% and later by 100%, sometimes I got spikes of
very bad performance, I guess it's GC or a merge

If indexing stops, it's not GC and not a merge. Then you did not
configure your indexing routine well enough to get it smoothly adjusted
to your system resources.
How do you index documents?

Question?

As I do not need to search the index while indexing, can I not do any
merges at all while indexing, and just do that at the end
my system is not real time at any sense, I am preparing a static index
to do read onl

Merging segments is not for searching, it is unevitable for he process
of creating an index. You can control the Lucene merge operation by

By reducing the merged segment size, you can lower the length of spikes
and take pressure from the heap if you have a small heap
(max_merged_segment = 2g instead of 5g for example)

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

David_MZ · June 20, 2013, 1:27pm

"If indexing stops, it's not GC and not a merge. Then you did not configure
your indexing routine well enough to get it smoothly adjusted to your
system resources.
How do you index documents?"

Indexing did not stop, it was just very slow, I use tire ruby gem to index
using bulk index operation

"By reducing the merged segment size, you can lower the length of spikes
and take pressure from the heap if you have a small heap
(max_merged_segment = 2g instead of 5g for example)"

What will happen if I set it to 500mb?

On Thu, Jun 20, 2013 at 4:22 PM, Jörg Prante joergprante@gmail.com wrote:

Am 20.06.13 15:14, schrieb David MZ:

The speed drops by 50% and later by 100%, sometimes I got spikes of very

bad performance, I guess it's GC or a merge

If indexing stops, it's not GC and not a merge. Then you did not
configure your indexing routine well enough to get it smoothly adjusted to
your system resources.
How do you index documents?

Question?

As I do not need to search the index while indexing, can I not do any
merges at all while indexing, and just do that at the end
my system is not real time at any sense, I am preparing a static index to
do read onl

Merging segments is not for searching, it is unevitable for he process of
creating an index. You can control the Lucene merge operation by
Elasticsearch Platform — Find real-time answers at scale | Elastic http://www.elasticsearch.org/guide/reference/index-modules/merge/

By reducing the merged segment size, you can lower the length of spikes
and take pressure from the heap if you have a small heap
(max_merged_segment = 2g instead of 5g for example)

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**rKjqMtgSqcE/unsubscribehttps://groups.google.com/d/topic/elasticsearch/rKjqMtgSqcE/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.com elasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.