ES running out of memory for bulk import

Elastic Stack Elasticsearch
1

Hi,

Im currently doing a large bulk import from mongodb into ES, the ES process
keeps on dying after a while, im assuming its running out of memory.

Ive got the following memory settings:
-Xms=512m
-Xmx=1500m

The logs are printing out this before it dies:

[2013-07-02 21:42:19,792][INFO ][monitor.jvm ] [Landslide]
[gc][ParNew][495][1351] duration [917ms], collections [1]/[1.4s], total
[917ms]/[1.1m], memory [405.2mb]->[375mb]/[1.4gb], all_pools {[Code Cache]
[6mb]->[6mb]/[48mb]}{[Par Eden Space] [38.8mb]->[3.4mb]/[133.1mb]}{[Par
Survivor Space] [7mb]->[6.1mb]/[16.6mb]}{[CMS Old Gen]
[359.5mb]->[365.4mb]/[1.3gb]}{[CMS Perm Gen] [41.3mb]->[41.3mb]/[166mb]}

Can anyone please help with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi,

The short "answer" is: it's 2013, can you give it a bigger Xmx? (and make
Xms that same value)

Otis

Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service
Search Analytics - Cloud Monitoring Tools & Services | Sematext

On Tuesday, July 2, 2013 9:53:21 PM UTC-4, Wesley Archbell wrote:

Hi,

Im currently doing a large bulk import from mongodb into ES, the ES
process keeps on dying after a while, im assuming its running out of memory.

Ive got the following memory settings:
-Xms=512m
-Xmx=1500m

The logs are printing out this before it dies:

[2013-07-02 21:42:19,792][INFO ][monitor.jvm ] [Landslide]
[gc][ParNew][495][1351] duration [917ms], collections [1]/[1.4s], total
[917ms]/[1.1m], memory [405.2mb]->[375mb]/[1.4gb], all_pools {[Code Cache]
[6mb]->[6mb]/[48mb]}{[Par Eden Space] [38.8mb]->[3.4mb]/[133.1mb]}{[Par
Survivor Space] [7mb]->[6.1mb]/[16.6mb]}{[CMS Old Gen]
[359.5mb]->[365.4mb]/[1.3gb]}{[CMS Perm Gen] [41.3mb]->[41.3mb]/[166mb]}

Can anyone please help with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

I add to Otis's answer that you could also decrease the number of docs per Bulk.
How many docs do you send each time?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 juil. 2013 à 07:15, Otis Gospodnetic otis.gospodnetic@gmail.com a écrit :

Hi,

The short "answer" is: it's 2013, can you give it a bigger Xmx? (and make Xms that same value)

Otis

Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service
Search Analytics - Cloud Monitoring Tools & Services | Sematext

On Tuesday, July 2, 2013 9:53:21 PM UTC-4, Wesley Archbell wrote:

Hi,

Im currently doing a large bulk import from mongodb into ES, the ES process keeps on dying after a while, im assuming its running out of memory.

Ive got the following memory settings:
-Xms=512m
-Xmx=1500m

The logs are printing out this before it dies:

[2013-07-02 21:42:19,792][INFO ][monitor.jvm ] [Landslide] [gc][ParNew][495][1351] duration [917ms], collections [1]/[1.4s], total [917ms]/[1.1m], memory [405.2mb]->[375mb]/[1.4gb], all_pools {[Code Cache] [6mb]->[6mb]/[48mb]}{[Par Eden Space] [38.8mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space] [7mb]->[6.1mb]/[16.6mb]}{[CMS Old Gen] [359.5mb]->[365.4mb]/[1.3gb]}{[CMS Perm Gen] [41.3mb]->[41.3mb]/[166mb]}

Can anyone please help with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Yep all good now, the box only had 912MB... made it 4Gb and gave ES 2Gb all
good now.

On Wednesday, July 3, 2013 3:15:47 PM UTC+10, Otis Gospodnetic wrote:

Hi,

The short "answer" is: it's 2013, can you give it a bigger Xmx? (and make
Xms that same value)

Otis

Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service
Search Analytics - Cloud Monitoring Tools & Services | Sematext

On Tuesday, July 2, 2013 9:53:21 PM UTC-4, Wesley Archbell wrote:

Hi,

Im currently doing a large bulk import from mongodb into ES, the ES
process keeps on dying after a while, im assuming its running out of memory.

Ive got the following memory settings:
-Xms=512m
-Xmx=1500m

The logs are printing out this before it dies:

[2013-07-02 21:42:19,792][INFO ][monitor.jvm ] [Landslide]
[gc][ParNew][495][1351] duration [917ms], collections [1]/[1.4s], total
[917ms]/[1.1m], memory [405.2mb]->[375mb]/[1.4gb], all_pools {[Code Cache]
[6mb]->[6mb]/[48mb]}{[Par Eden Space] [38.8mb]->[3.4mb]/[133.1mb]}{[Par
Survivor Space] [7mb]->[6.1mb]/[16.6mb]}{[CMS Old Gen]
[359.5mb]->[365.4mb]/[1.3gb]}{[CMS Perm Gen] [41.3mb]->[41.3mb]/[166mb]}

Can anyone please help with this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.