Logstash java.lang.OutOfMemoryError when reading yml file

[blueren (bluren)]

I'm currently running logstash with elastiflow configured within a docker container.

When logstash starts up, it attemps to read from ip_rep_basic.yml which is approx 48MB and 1559862 lines long.

Logstash seems to get stuck there for about 4 mins and then spits out the following:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Heap dump file created [1476784755 bytes in 60.679 secs]
[2018-08-25T10:36:52,253][ERROR][logstash.agent           ] Failed to execute action {:id=>:elastiflow, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<elastiflow>, action_result: false", :backtrace=>nil}
[2018-08-25T10:36:52,392][INFO ][logstash.inputs.metrics  ] Monitoring License OK
[2018-08-25T10:36:52,417][ERROR][org.logstash.Logstash    ] java.lang.OutOfMemoryError: Java heap space

I've zeroed in on this specific file since cutting down the number of lines in the file by a few thousand seems to make logstash work as intended.

I'm setting the heap size like so in the docker-compose.yml file. I've started with 512mb and gone up to 5gb but that does not seem to help. My current OS is Ubuntu 18.04 with 8Gb of ram and about 10 gb of swap space.

environment:
    - "ES_JAVA_OPTS=-Xms5g -Xmx5g"

Is there something else I'm missing?

[warkolm (Mark Walkom)]

It'd be worth raising an issue against https://github.com/logstash-plugins/logstash-filter-translate as there may be something we can do when working with such large translations dictionaries.

@rcowart have you seen this with other users of Elastiflow?

[rcowart (Rob Cowart)]

I just tested with the recommendations I have provided in the ElastiFlow readme and 4GB of JVM heap is enough to start it using 6.2.4, 6.3.2 and 6.4.0. However that is running on bare metal (my Macbook), not in a container. Also note that I am using the open source variants of Logstash. I notice from your log that you have monitoring enabled. I don't get enough value from monitoring to use it, and I don't know how much Heap it may be stealing.

@blueren please open issues on the ElastiFlow repo in the future. That will ensure that I see the issue more quickly.

@warkolm as the name of the file implies, this is the "basic" version of IP reputation tagging. My paying customers get a more complete/advanced IP reputation dictionary that varies in size between 5.5-6M entries and 110-135MB in size. A Logstash instance with 8GB Heap handles that dictionary and 35+ pipelines for flows and logs all working in harmony. So while that "basic" dictionary seems large, it isn't even close to those being run in production at many of my customers.

[blueren (bluren)]

Thanks for the quick check @rcowart. I did not raise this under elastiflow specifically since I felt that this is isolated with docker config only. As of now, I'm using the ip_rep_basic.yml from elastiflow 3.1.0 to get it working as I am not having any other clues on how to get this going.

UPDATE: I had to hit upon this SO question to realize my mistake.

I have to use LS_JAVA_OPTS for logstash instead of ES_JAVA_OPTS which I now realize is for elasticearch

Just tested it our and works perfectly. Good to close out this issue (Root cause - user error)

Thanks.