I tried to add a 1MB synonym file to an index which took more than 13 hours, after which I had to kill it because I couldn’t wait anymore (high cpu was bringing other services down).
1MB seems very small to me so I really can’t understand why it would take over 13 hours…
Hope something can be done about that or you have other tips to solve adding larger synonyms files.
From: Shay Banon [mailto:firstname.lastname@example.org]
Sent: zaterdag 4 juni 2011 18:35
Subject: Re: memory problems with larger synonyms file
Yea, I managed to simulate this. It does take time to process the synonym file. The other problem is that it is being processed twice, which I already fixed in master. Moreover, something that needs to be added is caching of the same synonym file for difference indices if they use the same one.
On Saturday, June 4, 2011 at 7:21 PM, Yannick Smits wrote:
Anyone been able to reproduce this one? Still unresolved.
Running on Windows 7 with Java 6 update 24 btw.
From: Yannick Smits [mailto:email@example.com]
Sent: dinsdag 31 mei 2011 11:39
Subject: memory problems with larger synonyms file
With this gist: https://gist.github.com/1000235 https://gist.github.com/1000235, when I run the index configuration the java.exe process goes to 30% CPU and memory usage grows to 400+MB’s, the index never gets created and it keeps the high CPU. The ES window tells me this kind of messages:
[2011-05-31 11:34:14,808][INFO ][monitor.jvm ] [test] [
gc][ConcurrentMarkSweep] took [5.7s]/[82ms], reclaimed [17.7mb], leaving [217.8mb] used, max [1gb]
[2011-05-31 11:34:32,865][INFO ][monitor.jvm ] [test] [
gc][ConcurrentMarkSweep] took [5.7s]/[85ms], reclaimed [30.4mb], leaving [221.3mb] used, max [1gb]
[2011-05-31 11:34:52,007][INFO ][monitor.jvm ] [test] [
gc][ConcurrentMarkSweep] took [5.7s]/[88ms], reclaimed [18.7mb], leaving [244.6mb] used, max [1gb]
When I try with smaller synonyms files it seems to work and create the index without problem.