[2013-06-28 11:17:24,697][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][256868][19712] duration [968ms], collections [1]/[2.2s], total
[968ms]/[4.3m], memory [675mb]->[651.1mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[813.6kb]/[33.3mb]}{[Par Survivor Space]
[157kb]->[1mb]/[4.1mb]}{[CMS Old Gen] [649.3mb]->[649.3mb]/[982.4mb]}{[CMS
Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:14,615][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][269713][19891] duration [792ms], collections [1]/[1.6s], total
[792ms]/[4.5m], memory [75mb]->[96.1mb]/[1019.8mb], all_pools {[Code Cache]
[7mb]->[7mb]/[48mb]}{[Par Eden Space] [4.4mb]->[698kb]/[33.3mb]}{[Par
Survivor Space] [301.8kb]->[827.6kb]/[4.1mb]}{[CMS Old Gen]
[70.2mb]->[94.6mb]/[982.4mb]}{[CMS Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:37,000][WARN ][monitor.jvm ] [Maestro]
[gc][ParNew][269730][19950] duration [2.2s], collections [2]/[2.3s], total
[2.2s]/[4.8m], memory [465.5mb]->[498.4mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[5.3mb]->[3.8mb]/[33.3mb]}{[Par Survivor Space]
[1mb]->[3.2mb]/[4.1mb]}{[CMS Old Gen] [459mb]->[491.3mb]/[982.4mb]}{[CMS
Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:39,004][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][269732][19952] duration [785ms], collections [1]/[1s], total
[785ms]/[4.8m], memory [544.4mb]->[527.3mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[31.4mb]->[2.8mb]/[33.3mb]}{[Par Survivor Space]
[795.9kb]->[4.1mb]/[4.1mb]}{[CMS Old Gen]
[512.1mb]->[520.4mb]/[982.4mb]}{[CMS Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
I understand that they probably have to do with GC, but what exactly are
the messages saying? Is there a problem and some action needs to be taken?
Because some are INFO and some are at WARN... ?
It means that garbage collection took longer than you'd expect (thresholds
are defined in your elasticsearch.yml) . Looking at your numbers, things
look odd. Normally it should be much much faster processing 544.4mb of
memory.
Is this a production machine? If so you probably want to allocate much more
memory to ES than the default 1GB. That settings is for development. Do
these log appear when there is also a very high query load? Something
prevents the GC from quickly finishing.
Cheers,
Boaz
On Tuesday, July 9, 2013 12:57:56 PM UTC+2, Kaushal wrote:
Hi All,
I have been getting these in my ES logs :
[2013-06-28 11:17:24,697][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][256868][19712] duration [968ms], collections [1]/[2.2s], total
[968ms]/[4.3m], memory [675mb]->[651.1mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[813.6kb]/[33.3mb]}{[Par Survivor Space]
[157kb]->[1mb]/[4.1mb]}{[CMS Old Gen] [649.3mb]->[649.3mb]/[982.4mb]}{[CMS
Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:14,615][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][269713][19891] duration [792ms], collections [1]/[1.6s], total
[792ms]/[4.5m], memory [75mb]->[96.1mb]/[1019.8mb], all_pools {[Code Cache]
[7mb]->[7mb]/[48mb]}{[Par Eden Space] [4.4mb]->[698kb]/[33.3mb]}{[Par
Survivor Space] [301.8kb]->[827.6kb]/[4.1mb]}{[CMS Old Gen]
[70.2mb]->[94.6mb]/[982.4mb]}{[CMS Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:37,000][WARN ][monitor.jvm ] [Maestro]
[gc][ParNew][269730][19950] duration [2.2s], collections [2]/[2.3s], total
[2.2s]/[4.8m], memory [465.5mb]->[498.4mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[5.3mb]->[3.8mb]/[33.3mb]}{[Par Survivor Space]
[1mb]->[3.2mb]/[4.1mb]}{[CMS Old Gen] [459mb]->[491.3mb]/[982.4mb]}{[CMS
Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:39,004][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][269732][19952] duration [785ms], collections [1]/[1s], total
[785ms]/[4.8m], memory [544.4mb]->[527.3mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[31.4mb]->[2.8mb]/[33.3mb]}{[Par Survivor Space]
[795.9kb]->[4.1mb]/[4.1mb]}{[CMS Old Gen]
[512.1mb]->[520.4mb]/[982.4mb]}{[CMS Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
I understand that they probably have to do with GC, but what exactly are
the messages saying? Is there a problem and some action needs to be taken?
Because some are INFO and some are at WARN... ?
It is a production machine - so yes I would seriously look into increasing
the allocated memory to ES, because I recently got a Heap space error as
well. Any ideas how much is enough - yes I know it all depends on the data
throughput and load, but say I have messages coming at 14 messages/ 5 sec,
with sometimes peaking to as much as 132 mes / sec ....?
On Tuesday, 9 July 2013 19:14:40 UTC+5:30, Boaz Leskes wrote:
Hi Kaushel,
It means that garbage collection took longer than you'd expect (thresholds
are defined in your elasticsearch.yml) . Looking at your numbers, things
look odd. Normally it should be much much faster processing 544.4mb of
memory.
Is this a production machine? If so you probably want to allocate much
more memory to ES than the default 1GB. That settings is for development.
Do these log appear when there is also a very high query load? Something
prevents the GC from quickly finishing.
Cheers,
Boaz
On Tuesday, July 9, 2013 12:57:56 PM UTC+2, Kaushal wrote:
Hi All,
I have been getting these in my ES logs :
[2013-06-28 11:17:24,697][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][256868][19712] duration [968ms], collections [1]/[2.2s], total
[968ms]/[4.3m], memory [675mb]->[651.1mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[813.6kb]/[33.3mb]}{[Par Survivor Space]
[157kb]->[1mb]/[4.1mb]}{[CMS Old Gen] [649.3mb]->[649.3mb]/[982.4mb]}{[CMS
Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:14,615][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][269713][19891] duration [792ms], collections [1]/[1.6s], total
[792ms]/[4.5m], memory [75mb]->[96.1mb]/[1019.8mb], all_pools {[Code Cache]
[7mb]->[7mb]/[48mb]}{[Par Eden Space] [4.4mb]->[698kb]/[33.3mb]}{[Par
Survivor Space] [301.8kb]->[827.6kb]/[4.1mb]}{[CMS Old Gen]
[70.2mb]->[94.6mb]/[982.4mb]}{[CMS Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:37,000][WARN ][monitor.jvm ] [Maestro]
[gc][ParNew][269730][19950] duration [2.2s], collections [2]/[2.3s], total
[2.2s]/[4.8m], memory [465.5mb]->[498.4mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[5.3mb]->[3.8mb]/[33.3mb]}{[Par Survivor Space]
[1mb]->[3.2mb]/[4.1mb]}{[CMS Old Gen] [459mb]->[491.3mb]/[982.4mb]}{[CMS
Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
[2013-06-28 14:52:39,004][INFO ][monitor.jvm ] [Maestro]
[gc][ParNew][269732][19952] duration [785ms], collections [1]/[1s], total
[785ms]/[4.8m], memory [544.4mb]->[527.3mb]/[1019.8mb], all_pools {[Code
Cache] [7mb]->[7mb]/[48mb]}{[Par Eden Space]
[31.4mb]->[2.8mb]/[33.3mb]}{[Par Survivor Space]
[795.9kb]->[4.1mb]/[4.1mb]}{[CMS Old Gen]
[512.1mb]->[520.4mb]/[982.4mb]}{[CMS Perm Gen] [37.8mb]->[37.8mb]/[166mb]}
I understand that they probably have to do with GC, but what exactly are
the messages saying? Is there a problem and some action needs to be taken?
Because some are INFO and some are at WARN... ?
From what I can deduce from your number, you run with default settings, and
you run ES in the cloud?
Note, in the cloud, you only have virtual machines. They show GC warnings
even though you even don't have GC problems! That's just because your
neighbors in the cloud put their memory pressure also on your virtual
machine, so your GC's take longer, and this causes warning messages. Boaz
is right, the GC statistics do not show that your workload demands for more
memory.
If you run ES locally, just check your RAM on this machine. Check if you
have other software running and move it another machine. Increase
ES_HEAP_SIZE to half of this size, and restart the node.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.