ES performance issues for 800G data per day

Elastic Stack Elasticsearch
1

Hi,
We plan to use ES to save the logs and search the logs.
800G logs or so will be produced per day.
Currently, we are running the performance test for long time.
We used 2 machines, 46 cores and 32G memory. The configuration is 2
shards, 1 replica, ES heap size is set to 20G. And only analyze the log
messages(omit norms and term frequency), all of other 5 fields are not
analyzed. And field cache type is set to soft, the max size is set to
10000, _source is set to true, _all is set to false.
We use Jmeter to simulate 600 thread to send the write request to the
ES cluster(write one log/per time) and at the same time, we use 10 thread
to query. The facet query are on all the fields.
Until now, we already indexed about 0.35 billion documents with
360G(720G)
During the test, we met the following the problems:
1. The cluster is easy to become yellow. Through ES head, we only can
see the node itself, the other node can't be seen.
After changing the multicast to unicast and restart the ES, the
problem is fixed, but after run 3-4 hours, the problem occurs again.
In fact, our system needs to run 724. This problem is really
unacceptable.
Is there any other way except restart ES which can fix this problem?
In addition, even if we restart the ES to fix the problem, it took
about 2 hours to fully recovery the cluster to green.

  2. sometimes, GC takes more than 1 hour. during these period, the ES

has no response for any request.
Is there any way to avoid so long time GC??

  3. The search performance is much more important than the indexing

for our system.
We plan to create a new index per month and use 5 ES servers(5 shards
+1 replica) in our product environment to handle 800G data per day.
Is this configure ok or not?? Is there any suggestion for the index
design and configuration?

  I'm almost crazy by these problem. Can anyone give me a hand. 
  Thank you very much!
2

Hey Jackie,

Sounds like a really interesting solution! When you say 800GB per day, is
that peak, or average usage, and does that include a replication factor of
some sort? in addition, does that maintain a steady rate of ingest, or
would it be bursty throughout the day? Can you perhaps increase the number
of nodes for your testing? I'd personally run this sort of ingest rate
(assuming that's per day) at more like 10+ nodes, easily, with
significantly more shards, and probably 3 replicas, and while you are
increasing the nodes in production, it'd be hard to get any sort of
reasonable testing with 2 nodes IMHO.

When you say 2 machines with 24 cores, and 32GB of RAM, you don't seem to
mention the disk layout, which I'd guess with that sort of data input would
be significantly more important.

Do you employ _source compression? If so, do you use thresholds? You could
certainly increase the indexes to daily indexes, and use aliases to create
monthly indexing.The big thing that makes ES bad-ass is it's distributed
nature, the more performance you need, the more nodes you throw at it...
Here's some information to add to your search :

Do you by any chance have some information that you provide on what the
average document looks like, or perhaps what the average search looks like?

Patrick

patrick eefy net

On Tue, May 22, 2012 at 6:33 AM, jackiedong jackiedong168@gmail.com wrote:

Hi,
We plan to use ES to save the logs and search the logs.
800G logs or so will be produced per day.
Currently, we are running the performance test for long time.
We used 2 machines, 46 cores and 32G memory. The configuration is 2
shards, 1 replica, ES heap size is set to 20G. And only analyze the log
messages(omit norms and term frequency), all of other 5 fields are not
analyzed. And field cache type is set to soft, the max size is set to
10000, _source is set to true, _all is set to false.
We use Jmeter to simulate 600 thread to send the write request to the
ES cluster(write one log/per time) and at the same time, we use 10 thread
to query. The facet query are on all the fields.
Until now, we already indexed about 0.35 billion documents with
360G(720G)
During the test, we met the following the problems:
1. The cluster is easy to become yellow. Through ES head, we only can
see the node itself, the other node can't be seen.
After changing the multicast to unicast and restart the ES, the
problem is fixed, but after run 3-4 hours, the problem occurs again.
In fact, our system needs to run 724. This problem is really
unacceptable.
Is there any other way except restart ES which can fix this problem?
In addition, even if we restart the ES to fix the problem, it took
about 2 hours to fully recovery the cluster to green.

  2. sometimes, GC takes more than 1 hour. during these period, the ES

has no response for any request.
Is there any way to avoid so long time GC??

  3. The search performance is much more important than the indexing

for our system.
We plan to create a new index per month and use 5 ES servers(5
shards +1 replica) in our product environment to handle 800G data per day.
Is this configure ok or not?? Is there any suggestion for the index
design and configuration?

  I'm almost crazy by these problem. Can anyone give me a hand.
  Thank you very much!
3 
  2. sometimes, GC takes more than 1 hour. during these period,

the ES has no response for any request.
Is there any way to avoid so long time GC??

Have you disabled swap? Swap kills the JVM's garbage collection. You
can either disable it outright, or use boostrap.mlockall to prevent the
heap from being swapped out.

Google this group for posts about mlockall

4

Hi, Patrick:
Thank you very much for your response.
In the production environment, the average usage will be 800G per day
without replica.
But at this moment, we just do some initial test to see if ES really
can be adopted. So at this moment, the data is not very huge, we just use
600 thread in jmeter to send write request, per log is about 2k. The disk
of test machine is just 1.2 T.
We don't use _compress yet. The search performance is much more
important than the disk space saving.
The log example is as below:
{ "projectName" : "${appname}",
"projectVersion" : "1.0",
"logType": "${logtype}",
"logSource": "access log",
"logLevel":"1",
"logTime":"2012-${month}-${day}T${hour}:${min}:${sec}",
"host":"${__machineIP}",
"body": "${__machineIP} - - GET /${appname}/${logtype}/log HTTP/1.1
clothes simple comfort and Beggar sareoftenmi sunder stoodas people who
have support in the form of pennies and loose change To these people
feeling that they have given up any hope of getting their feet back on the
ground is anormal response if they were ask edup close and personal While
sharing an ickelora quarter may goal on gway for them it is hard to believe
the people would simply lie over and die Butto some people in such an
unfortun a tesituationitis more than simple surrenderbut anothe raspectof
personal proportions that have led them to lose hope and live the",
"fields": "${fieldvalue}"
}
The performance seems ok if ES is running stable.
But the real problem is ES is not stable. We just inserted 300G or so
data, it often can't work correctly.
1) The cluster is broken often. Two nodes can't see each other.
Is there any way to recover it except to restart ES?
2) GC for long time, sometimes more than 1 hour.
Is there any way to avoid it?
By the way, we already set boostrap.mlockall to true.
Thanks.

在 2012年5月22日星期二UTC+8下午11时29分33秒，Patrick Ancillotti写道：

Hey Jackie,

Sounds like a really interesting solution! When you say 800GB per day, is
that peak, or average usage, and does that include a replication factor of
some sort? in addition, does that maintain a steady rate of ingest, or
would it be bursty throughout the day? Can you perhaps increase the number
of nodes for your testing? I'd personally run this sort of ingest rate
(assuming that's per day) at more like 10+ nodes, easily, with
significantly more shards, and probably 3 replicas, and while you are
increasing the nodes in production, it'd be hard to get any sort of
reasonable testing with 2 nodes IMHO.

When you say 2 machines with 24 cores, and 32GB of RAM, you don't seem to
mention the disk layout, which I'd guess with that sort of data input would
be significantly more important.

Do you employ _source compression? If so, do you use thresholds? You could
certainly increase the indexes to daily indexes, and use aliases to create
monthly indexing.The big thing that makes ES bad-ass is it's distributed
nature, the more performance you need, the more nodes you throw at it...
Here's some information to add to your search :

GitHub - elastic/logstash: Logstash - transport and process your logs, events, or other data

Do you by any chance have some information that you provide on what the
average document looks like, or perhaps what the average search looks like?

Patrick

Patrick Ancillotti - New York | about.me
patrick eefy net

On Tue, May 22, 2012 at 6:33 AM, jackiedong jackiedong168@gmail.comwrote:

Hi,
We plan to use ES to save the logs and search the logs.
800G logs or so will be produced per day.
Currently, we are running the performance test for long time.
We used 2 machines, 46 cores and 32G memory. The configuration is 2
shards, 1 replica, ES heap size is set to 20G. And only analyze the log
messages(omit norms and term frequency), all of other 5 fields are not
analyzed. And field cache type is set to soft, the max size is set to
10000, _source is set to true, _all is set to false.
We use Jmeter to simulate 600 thread to send the write request to
the ES cluster(write one log/per time) and at the same time, we use 10
thread to query. The facet query are on all the fields.
Until now, we already indexed about 0.35 billion documents with
360G(720G)
During the test, we met the following the problems:
1. The cluster is easy to become yellow. Through ES head, we only
can see the node itself, the other node can't be seen.
After changing the multicast to unicast and restart the ES, the
problem is fixed, but after run 3-4 hours, the problem occurs again.
In fact, our system needs to run 724. This problem is really
unacceptable.
Is there any other way except restart ES which can fix this problem?
In addition, even if we restart the ES to fix the problem, it took
about 2 hours to fully recovery the cluster to green.

  2. sometimes, GC takes more than 1 hour. during these period, the

ES has no response for any request.
Is there any way to avoid so long time GC??

  3. The search performance is much more important than the indexing

for our system.
We plan to create a new index per month and use 5 ES servers(5
shards +1 replica) in our product environment to handle 800G data per day.
Is this configure ok or not?? Is there any suggestion for the index
design and configuration?

  I'm almost crazy by these problem. Can anyone give me a hand. 
  Thank you very much!
5

Hi, Clinton:
We already set boostrap.mlockall to true to prevent the heap from
being swapped out.

在 2012年5月22日星期二UTC+8下午11时43分04秒，Clinton Gormley写道：

  2. sometimes, GC takes more than 1 hour. during these period,

the ES has no response for any request.
Is there any way to avoid so long time GC??

Have you disabled swap? Swap kills the JVM's garbage collection. You
can either disable it outright, or use boostrap.mlockall to prevent the
heap from being swapped out.

Google this group for posts about mlockall

6

Hi, Patrick:
Thank you for your response.
In the product environment, the average data will be 800G per day
without replica.
But currently, we just do some initial test to see if we really can
adopt ES as the solution.
The test machine only have 1.2 T disk.
We haven't set compression for _source. The search performance is much
more important than the disk save.
The sample data we insert is as following:
{ "projectName" : "${appname}",
"projectVersion" : "1.0",
"logType": "${logtype}",
"logSource": "access log",
"logLevel":"1",
"logTime":"2012-${month}-${day}T${hour}:${min}:${sec}",
"host":"${__machineIP}",
"body": "${__machineIP} - - GET /${appname}/${logtype}/log HTTP/1.1
clothes simple comfort and Beggar sareoftenmi sunder stoodas people who
have support in the form of pennies and loose change To these people
feeling that they have given up any hope of getting their feet back on the
ground is anormal response if they were ask edup close and personal While
sharing an ickelora quarter may goal on gway for them it is hard to believe
the people would simply lie over and die Butto some people in such an
unfortun a tesituationitis more than simple surrenderbut anothe raspectof
personal proportions that have led them to lose hope and live the",
"fields": "${fieldvalue}"
}
The performance seems ok if ES is running stable.
But the real problem is ES is not stable. We just use jmeter to
simulate 300 thread to send write request to the ES, and the data size just
reaches to about 300G.
1) The cluster is very easy to be broken. The nodes in the cluster
can't see each other.
Is there any way to bring the cluster back to green except
restart the ES?

   2) GC sometimes took more than 1 hour. 
      Is there any way to avoid it? I think our test load is not so

much. I'm wondering why such thing happens.

在 2012年5月22日星期二UTC+8下午11时29分33秒，Patrick Ancillotti写道：

Hey Jackie,

Sounds like a really interesting solution! When you say 800GB per day, is
that peak, or average usage, and does that include a replication factor of
some sort? in addition, does that maintain a steady rate of ingest, or
would it be bursty throughout the day? Can you perhaps increase the number
of nodes for your testing? I'd personally run this sort of ingest rate
(assuming that's per day) at more like 10+ nodes, easily, with
significantly more shards, and probably 3 replicas, and while you are
increasing the nodes in production, it'd be hard to get any sort of
reasonable testing with 2 nodes IMHO.

When you say 2 machines with 24 cores, and 32GB of RAM, you don't seem to
mention the disk layout, which I'd guess with that sort of data input would
be significantly more important.

Do you employ _source compression? If so, do you use thresholds? You could
certainly increase the indexes to daily indexes, and use aliases to create
monthly indexing.The big thing that makes ES bad-ass is it's distributed
nature, the more performance you need, the more nodes you throw at it...
Here's some information to add to your search :

GitHub - elastic/logstash: Logstash - transport and process your logs, events, or other data

Do you by any chance have some information that you provide on what the
average document looks like, or perhaps what the average search looks like?

Patrick

Patrick Ancillotti - New York | about.me
patrick eefy net

On Tue, May 22, 2012 at 6:33 AM, jackiedong jackiedong168@gmail.comwrote:

Hi,
We plan to use ES to save the logs and search the logs.
800G logs or so will be produced per day.
Currently, we are running the performance test for long time.
We used 2 machines, 46 cores and 32G memory. The configuration is 2
shards, 1 replica, ES heap size is set to 20G. And only analyze the log
messages(omit norms and term frequency), all of other 5 fields are not
analyzed. And field cache type is set to soft, the max size is set to
10000, _source is set to true, _all is set to false.
We use Jmeter to simulate 600 thread to send the write request to
the ES cluster(write one log/per time) and at the same time, we use 10
thread to query. The facet query are on all the fields.
Until now, we already indexed about 0.35 billion documents with
360G(720G)
During the test, we met the following the problems:
1. The cluster is easy to become yellow. Through ES head, we only
can see the node itself, the other node can't be seen.
After changing the multicast to unicast and restart the ES, the
problem is fixed, but after run 3-4 hours, the problem occurs again.
In fact, our system needs to run 724. This problem is really
unacceptable.
Is there any other way except restart ES which can fix this problem?
In addition, even if we restart the ES to fix the problem, it took
about 2 hours to fully recovery the cluster to green.

  2. sometimes, GC takes more than 1 hour. during these period, the

ES has no response for any request.
Is there any way to avoid so long time GC??

  3. The search performance is much more important than the indexing

for our system.
We plan to create a new index per month and use 5 ES servers(5
shards +1 replica) in our product environment to handle 800G data per day.
Is this configure ok or not?? Is there any suggestion for the index
design and configuration?

  I'm almost crazy by these problem. Can anyone give me a hand. 
  Thank you very much!
7

On Tue, 2012-05-22 at 17:25 -0700, jackiedong wrote:

Hi, Clinton:
We already set boostrap.mlockall to true to prevent the heap from
being swapped out.

Are you sure it is being applied? I press this point because if it isn't
setup correctly, it won't work.

When you start up ES check the heap size (you can use bigdesk) and make
sure that it is whatever you have set ES_MAX_MEM to.

And keep an eye on swap - is it growing?

Try just turning swap off completely. If that helps, then you probably
don't have the memory locked correctly.

I'm harping on this because the only times I've seen GC take so long is
when swap is involved.

You specifically mentioned long GC times, but the other thing which can
cause long delays are merges. Merges can be very IO intensive and
essentially bring a box to a standstill.

One of my indexes is very write intensive, and I reduced the max segment
size from 5GB to 1GB to reduce the long merges. The downside of that is
that searching is slower.

clint

8

Hi, Clinton:
Thank you very much for your help. I will check it later.
By the way, have you ever met the cluster's problem? I mean the nodes
in the cluster can't see each other which cause the cluster to become
yellow. And without restart the ES, the cluster can't come back to green
anymore although each node can still work separately.

-Regards-
-Jackie-

在 2012年5月23日星期三UTC+8下午4时48分48秒，Clinton Gormley写道：

On Tue, 2012-05-22 at 17:25 -0700, jackiedong wrote:

Hi, Clinton:
We already set boostrap.mlockall to true to prevent the heap from
being swapped out.

Are you sure it is being applied? I press this point because if it isn't
setup correctly, it won't work.

When you start up ES check the heap size (you can use bigdesk) and make
sure that it is whatever you have set ES_MAX_MEM to.

And keep an eye on swap - is it growing?

Try just turning swap off completely. If that helps, then you probably
don't have the memory locked correctly.

I'm harping on this because the only times I've seen GC take so long is
when swap is involved.

You specifically mentioned long GC times, but the other thing which can
cause long delays are merges. Merges can be very IO intensive and
essentially bring a box to a standstill.

One of my indexes is very write intensive, and I reduced the max segment
size from 5GB to 1GB to reduce the long merges. The downside of that is
that searching is slower.

clint

9

I'd guess that the hardware you're using is quite under powered for the
tests you're running. 600 threads pounding the servers and since you have a
replica it's like there is only 1 server to handle all the load. Have you
been monitoring the JVM while doing the tests? The cluster nodes not being
able to see each other can be due to load. For example, if the JVM is short
of memory, it can struggle quite a lot and not respond to queries, causing
the other servers in the cluster to conclude that it's down. I'd start with
a lower rate and monitor the server and JVM performance to see whether it's
running out of juice. Cluster broken problem is likely result of resource
shortage.
Couple of other things:

  • If you will have 800GBs per day, you should create an index per day
    rather than per month, especially the queries you'll run have a time
    component. Optimizing the search performance is much easier if you know
    what type of queries you will run and design your indices accordingly.
  • Are you using the bulk API?

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Tue, May 22, 2012 at 8:22 PM, jackiedong jackiedong168@gmail.com wrote:

Hi, Patrick:
Thank you very much for your response.
In the production environment, the average usage will be 800G per day
without replica.
But at this moment, we just do some initial test to see if ES really
can be adopted. So at this moment, the data is not very huge, we just use
600 thread in jmeter to send write request, per log is about 2k. The disk
of test machine is just 1.2 T.
We don't use _compress yet. The search performance is much more
important than the disk space saving.
The log example is as below:
{ "projectName" : "${appname}",
"projectVersion" : "1.0",
"logType": "${logtype}",
"logSource": "access log",
"logLevel":"1",
"logTime":"2012-${month}-${day}T${hour}:${min}:${sec}",
"host":"${__machineIP}",
"body": "${__machineIP} - - GET /${appname}/${logtype}/log HTTP/1.1
clothes simple comfort and Beggar sareoftenmi sunder stoodas people who
have support in the form of pennies and loose change To these people
feeling that they have given up any hope of getting their feet back on the
ground is anormal response if they were ask edup close and personal While
sharing an ickelora quarter may goal on gway for them it is hard to believe
the people would simply lie over and die Butto some people in such an
unfortun a tesituationitis more than simple surrenderbut anothe raspectof
personal proportions that have led them to lose hope and live the",
"fields": "${fieldvalue}"
}
The performance seems ok if ES is running stable.
But the real problem is ES is not stable. We just inserted 300G or so
data, it often can't work correctly.
1) The cluster is broken often. Two nodes can't see each other.
Is there any way to recover it except to restart ES?
2) GC for long time, sometimes more than 1 hour.
Is there any way to avoid it?
By the way, we already set boostrap.mlockall to true.
Thanks.

在 2012年5月22日星期二UTC+8下午11时29分33秒，Patrick Ancillotti写道：

Hey Jackie,

Sounds like a really interesting solution! When you say 800GB per day, is
that peak, or average usage, and does that include a replication factor of
some sort? in addition, does that maintain a steady rate of ingest, or
would it be bursty throughout the day? Can you perhaps increase the number
of nodes for your testing? I'd personally run this sort of ingest rate
(assuming that's per day) at more like 10+ nodes, easily, with
significantly more shards, and probably 3 replicas, and while you are
increasing the nodes in production, it'd be hard to get any sort of
reasonable testing with 2 nodes IMHO.

When you say 2 machines with 24 cores, and 32GB of RAM, you don't seem to
mention the disk layout, which I'd guess with that sort of data input would
be significantly more **important.

Do you employ _source compression? If so, do you use thresholds? You
could certainly increase the indexes to daily indexes, and use aliases to
create monthly indexing.The big thing that makes ES bad-ass is it's
distributed nature, the more performance you need, the more nodes you throw
at it... Here's some information to add to your search :

https://github.com/logstash/**logstash/wiki/Elasticsearch-**
Storage-Optimizationhttps://github.com/logstash/logstash/wiki/Elasticsearch-Storage-Optimization

Do you by any chance have some information that you provide on what the
average document looks like, or perhaps what the average search looks like?

Patrick
------------------------------**----------
http://about.me/patrick.**ancillotti http://about.me/patrick.ancillotti
patrick eefy net

On Tue, May 22, 2012 at 6:33 AM, jackiedong jackiedong168@gmail.comwrote:

Hi,
We plan to use ES to save the logs and search the logs.
800G logs or so will be produced per day.
Currently, we are running the performance test for long time.
We used 2 machines, 46 cores and 32G memory. The configuration is
2 shards, 1 replica, ES heap size is set to 20G. And only analyze the log
messages(omit norms and term frequency), all of other 5 fields are not
analyzed. And field cache type is set to soft, the max size is set to
10000, _source is set to true, _all is set to false.
We use Jmeter to simulate 600 thread to send the write request to
the ES cluster(write one log/per time) and at the same time, we use 10
thread to query. The facet query are on all the fields.
Until now, we already indexed about 0.35 billion documents with
360G(720G)
During the test, we met the following the problems:
1. The cluster is easy to become yellow. Through ES head, we only
can see the node itself, the other node can't be seen.
After changing the multicast to unicast and restart the ES, the
problem is fixed, but after run 3-4 hours, the problem occurs again.
In fact, our system needs to run 724. This problem is really
unacceptable.
Is there any other way except restart ES which can fix this
problem?
In addition, even if we restart the ES to fix the problem, it took
about 2 hours to fully recovery the cluster to green.

  2. sometimes, GC takes more than 1 hour. during these period, the

ES has no response for any request.
Is there any way to avoid so long time GC??

  3. The search performance is much more important than the indexing

for our system.
We plan to create a new index per month and use 5 ES servers(5
shards +1 replica) in our product environment to handle 800G data per day.
Is this configure ok or not?? Is there any suggestion for the
index design and configuration?

  I'm almost crazy by these problem. Can anyone give me a hand.
  Thank you very much!