Logstash won't start due to memory heap issue

Elastic Stack Logstash
1

Hello,

I have a instance running logstsh 6.8.2 with Java 11.0.19 and JDK of 64Bits, I changed the filter file do add conditions and terraform logstash to the correct environment. After that logstash refused to start giving me a Invalid initial heap size -Xms2g.

I have 5G of free space on that and I don't have anyrhing running on my machine that is occupying the memory.

As of now I have tried changing the heap size to several values, obeying the fact that I have in total 7G and 5G of free space and nothing solved this issue.

I looked thru the forum and tryed the suggestions and did not fix it, what do youi guys suggest I do?

2

What command line is Java being started with and exactly what error message do you get?

3

Hi, I’m starting logstash using the following script:

sudo systemctl start logastash

And the error I get is:

Invalid minimal heap size Xms2g

4

Are you absolutely without-a-doubt 100% certain that you are running a 64-bit JVM and not a 32-bit JVM?

I suggest dropping the heap size down to 512 MB and then reviewing the logs, which should include full details of the Java version being started. It doesn't matter if you cannot run your pipeline in 512 MB, you just need to get the VM up far enough to log those details. It's OK if it crashes after logging that.

5

Hi, yes. I ran the script java --version and I get the 64 Bits response.

I already did this 512 MB try and it provided me the same issue of memory heap size. No matter what value I add to the jvm options it is giving me the heap error.

The logstash plain log does't update since the memory error and the syslog keeps showing the memory heap issue for each try that I did for the memory.

I'm really lost :frowning:

6

To add more ideas:

  • make a simple .conf which has input tcp or random, nothing in filter. Something like this:
input {
  generator {
       message => "Test message"
	   count => 1 }
}
filter {
}
output {
    stdout { codec => rubydebug{} }
}
  • Badger suggested 512 MB, however 1 GB is enough in the most cases when you have file or beats inputs without complex JSON/XML transformation. On large data set, LS with start and will notice how consumes memory and slow down processing
  • Use debug/trace log for more details
  • Check do you have enough memory on the host. Sometimes there is a basic configuration, 2-4 GB memory with ES and Kib

How long is the message? 1 KB, 2 KB, more? Which plugins are you using in the filter?

7

Can you share the exact log error you are getting? Try to start it again and copy the line logs generated.

Also, share your logstash.yml and jvm.options as well.

8

Hi , sure here is what you asked:Sep 05 12:05:18 ip-IPsystemd[1]: Started logstash.
Sep 04 19:53:08 ip-IPsystemd[1]: Failed to start logstash.
Sep 04 19:53:08 ip-IPsystemd[1]: logstash.service: Failed with result 'exit-code'.
Sep 04 19:53:08 ip-IPsystemd[1]: logstash.service: Start request repeated too quickly.
Sep 04 19:53:08 ip-IPsystemd[1]: Stopped logstash.
Sep 04 19:53:08 ip-IPsystemd[1]: logstash.service: Scheduled restart job, restart counter is at 5.
Sep 04 19:53:08 ip-IPsystemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Sep 04 19:53:07 ip-IPsystemd[1]: logstash.service: Failed with result 'exit-code'.
Sep 04 19:53:07 ip-IPsystemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Sep 04 19:53:07 ip-IPlogstash[10934]: Error: A fatal exception has occurred. Program will exit.
Sep 04 19:53:07 ip-IPlogstash[10934]: Error: Could not create the Java Virtual Machine.
Sep 04 19:53:07 ip-IPlogstash[10934]: Invalid initial heap size: -Xms2g
Sep 04 19:53:07 ip-IPsystemd[1]: Started logstash.
Sep 04 19:53:07 ip-IPsystemd[1]: Stopped logstash.
Sep 04 19:53:07 ip-IPsystemd[1]: logstash.service: Scheduled restart job, restart counter is at 4.

logstash.yml

path.data: /var/lib/logstash
queue.type: persisted
path.queue: /var/logstash/queue
queue.page_capacity: 64mb
queue.max_bytes: 4096mb
queue.checkpoint.writes: 1024
path.logs: /var/log/logstash
config.reload.automatic: true
config.reload.interval: 60s
pipeline.workers: 2
pipeline.batch.size: 250

jmv.options file is all stardart, the only change is on the heap size:

JVM configuration

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

################################################################

Expert settings

################################################################

All settings below this section are considered

expert settings. Don't tamper with them unless

Thanks

9

Have you checked the journal?
journalctl -u logstash.service -n 100

10

Can you run free -h in this server and share the result?

Also, can you change g to G and test again?

-Xms2G
-Xmx2G
11

Hi everyone, I have found what was the issue.

ran a dos2unix script on my jvm.options and fixed the issue starting logstash again.

Thank you for all the suggestions, it was very helpful