Memory problems with larger synonyms file

Elastic Stack Elasticsearch
1

With this gist: https://gist.github.com/1000235
https://gist.github.com/1000235, when I run the index configuration the
java.exe process goes to 30% CPU and memory usage grows to 400+MB's, the
index never gets created and it keeps the high CPU. The ES window tells me
this kind of messages:

[2011-05-31 11:34:14,808][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][25] took [5.7s]/[82ms], reclaimed [17.7mb], leaving
[217.8mb] used, max [1gb]

[2011-05-31 11:34:32,865][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][26] took [5.7s]/[85ms], reclaimed [30.4mb], leaving
[221.3mb] used, max [1gb]

[2011-05-31 11:34:52,007][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][27] took [5.7s]/[88ms], reclaimed [18.7mb], leaving
[244.6mb] used, max [1gb]

When I try with smaller synonyms files it seems to work and create the index
without problem.

Thanks,

Yannick

2

Anyone been able to reproduce this one? Still unresolved.

Running on Windows 7 with Java 6 update 24 btw.

Thanks,

Yannick

From: Yannick Smits [mailto:mailinglists@goyaweb.nl]
Sent: dinsdag 31 mei 2011 11:39
To: users@elasticsearch.com
Subject: memory problems with larger synonyms file

With this gist: https://gist.github.com/1000235
https://gist.github.com/1000235, when I run the index configuration the
java.exe process goes to 30% CPU and memory usage grows to 400+MB's, the
index never gets created and it keeps the high CPU. The ES window tells me
this kind of messages:

[2011-05-31 11:34:14,808][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][25] took [5.7s]/[82ms], reclaimed [17.7mb], leaving
[217.8mb] used, max [1gb]

[2011-05-31 11:34:32,865][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][26] took [5.7s]/[85ms], reclaimed [30.4mb], leaving
[221.3mb] used, max [1gb]

[2011-05-31 11:34:52,007][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][27] took [5.7s]/[88ms], reclaimed [18.7mb], leaving
[244.6mb] used, max [1gb]

When I try with smaller synonyms files it seems to work and create the index
without problem.

Thanks,

Yannick

3

Heya,

Yea, I managed to simulate this. It does take time to process the synonym file. The other problem is that it is being processed twice, which I already fixed in master. Moreover, something that needs to be added is caching of the same synonym file for difference indices if they use the same one.

-shay.banon

On Saturday, June 4, 2011 at 7:21 PM, Yannick Smits wrote:

Anyone been able to reproduce this one? Still unresolved.
Running on Windows 7 with Java 6 update 24 btw.

Thanks,
Yannick

From: Yannick Smits [mailto:mailinglists@goyaweb.nl]
Sent: dinsdag 31 mei 2011 11:39
To: users@elasticsearch.com (mailto:users@elasticsearch.com)
Subject: memory problems with larger synonyms file

With this gist: 1000235’s gists · GitHub, when I run the index configuration the java.exe process goes to 30% CPU and memory usage grows to 400+MB’s, the index never gets created and it keeps the high CPU. The ES window tells me this kind of messages:

[2011-05-31 11:34:14,808][INFO ][monitor.jvm ] [test] [
gc][ConcurrentMarkSweep][25] took [5.7s]/[82ms], reclaimed [17.7mb], leaving [217.8mb] used, max [1gb]
[2011-05-31 11:34:32,865][INFO ][monitor.jvm ] [test] [
gc][ConcurrentMarkSweep][26] took [5.7s]/[85ms], reclaimed [30.4mb], leaving [221.3mb] used, max [1gb]
[2011-05-31 11:34:52,007][INFO ][monitor.jvm ] [test] [
gc][ConcurrentMarkSweep][27] took [5.7s]/[88ms], reclaimed [18.7mb], leaving [244.6mb] used, max [1gb]

When I try with smaller synonyms files it seems to work and create the index without problem.

Thanks,
Yannick

4

I tried to add a 1MB synonym file to an index which took more than 13 hours, after which I had to kill it because I couldn’t wait anymore (high cpu was bringing other services down).

1MB seems very small to me so I really can’t understand why it would take over 13 hours…

Hope something can be done about that or you have other tips to solve adding larger synonyms files.

Thanks,
Yannick

From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: zaterdag 4 juni 2011 18:35
To: users@elasticsearch.com
Subject: Re: memory problems with larger synonyms file

Heya,

Yea, I managed to simulate this. It does take time to process the synonym file. The other problem is that it is being processed twice, which I already fixed in master. Moreover, something that needs to be added is caching of the same synonym file for difference indices if they use the same one.

-shay.banon

On Saturday, June 4, 2011 at 7:21 PM, Yannick Smits wrote:

Anyone been able to reproduce this one? Still unresolved.

Running on Windows 7 with Java 6 update 24 btw.

Thanks,

Yannick

From: Yannick Smits [mailto:mailinglists@goyaweb.nl]
Sent: dinsdag 31 mei 2011 11:39
To: users@elasticsearch.com
Subject: memory problems with larger synonyms file

With this gist: https://gist.github.com/1000235 https://gist.github.com/1000235, when I run the index configuration the java.exe process goes to 30% CPU and memory usage grows to 400+MB’s, the index never gets created and it keeps the high CPU. The ES window tells me this kind of messages:

[2011-05-31 11:34:14,808][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][25] took [5.7s]/[82ms], reclaimed [17.7mb], leaving [217.8mb] used, max [1gb]

[2011-05-31 11:34:32,865][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][26] took [5.7s]/[85ms], reclaimed [30.4mb], leaving [221.3mb] used, max [1gb]

[2011-05-31 11:34:52,007][INFO ][monitor.jvm ] [test] [

gc][ConcurrentMarkSweep][27] took [5.7s]/[88ms], reclaimed [18.7mb], leaving [244.6mb] used, max [1gb]

When I try with smaller synonyms files it seems to work and create the index without problem.

Thanks,

Yannick