Hi,
I use filebeat to send logs to logstash who is running in a docker's container on ubuntu.
Since the gathered log files format was modified, i have a lot of grokparse failure. I don't know if there is a relation but know, my logstash server crashes.
I have this error :
Unhandled Java exception: java.lang.BootstrapMethodError: call site initialization exception
java.lang.BootstrapMethodError: call site initialization exception
[.....]
Caused by:
[2018-07-26T14:08:42,862][WARN ][io.netty.channel.AbstractChannelHandlerContext] The exceptionCaught() event that was failed to submit was:
java.util.concurrent.RejectedExecutionException: event executor terminated
[...]
[2018-07-26T14:08:45,241][WARN ][io.netty.channel.AbstractChannelHandlerContext] Failed to submit an exceptionCaught() event.
java.util.concurrent.RejectedExecutionException: event executor terminated
[...]
[2018-07-26T15:40:15,319][WARN ][logstash.filters.json ] Error parsing json {:source=>"json_column", :raw=>"base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5[...]AX9KtJ4U09Rwv6VMnhewH8P6VctdBs4Xyq/pWkthCBgCl+ww5zinCxh9KclnEvIWphAg7UxrZG6037HF6Uv2SL0pPscXpSGyi9KQ2MJ7Uz+zYPSl/s6D+7SjT4B/DThYQf3amS2iRcBRX//2Q==\"}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'base64': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5[...]AX9KtJ4U09Rwv6VMnhewH8P6VctdBs4Xyq/pWkthCBgCl+ww5zinCxh9KclnEvIWphAg7UxrZG6037HF6Uv2SL0pPscXpSGyi9KQ2MJ7Uz+zYPSl/s6D+7SjT4B/DThYQf3amS2iRcBRX//2Q=="}"; line: 1, column: 8]>}
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Unable to create java_pid1.hprof: File exists
here are my logstash container stats :
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8b377d4515b8 logstash 78.27% 1.279GiB / 11.73GiB 10.90% 9.57MB / 22.5MB 19.6MB / 5.2MB 78
Have you an idea on how to fix this ?
Thx a lot for your help.
BR,
Khaled
The JVM ran out of memory. Have you tried increasing the JVM heap?
Thank you Magnus.
I'm running logstash in a docker container on an ubuntu distribution, and i'm really not comfortable with this 
So, i would be glad if you give me the command line to increase the JVM heap memory through the docker container.
Thanks à lot for your help.
Which Docker image are you using?
i don't know which image was used. here is the output for a docker ps -a command :
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7fa5af7743fe java "java -Xmx4096m" 3 hours ago Exited (1) 3 hours ago zen_payne
be7856979552 java "-Xmx4096m" 3 hours ago Created suspicious_colden
9c154454b4e9 java "-Xmx4096m" 3 hours ago Created loving_lumiere
d6384d68cbb8 java "java -Xmx4096m" 3 hours ago Exited (1) 3 hours ago elastic_lamport
c01fd3e78163 java "java -XshowSettings…" 3 hours ago Exited (0) 3 hours ago musing_blackwell
974b8336261d java "java -XshowSettings…" 3 hours ago Exited (1) 3 hours ago goofy_austin
c19469e87258 java "java -Xmx10000m -X2…" 3 hours ago Exited (1) 3 hours ago determined_villani
c526296033af java "-Xmx10000m -X2000m …" 3 hours ago Created zealous_shaw
9590a1890b17 java "java -XshowSettings…" 3 hours ago Exited (0) 3 hours ago festive_ptolemy
05d2f68696cc logstash "/docker-entrypoint.…" 10 days ago Exited (0) 10 days ago relaxed_agnesi
8b377d4515b8 docker.elastic.co/logstash/logstash-oss:6.1.4 "/usr/local/bin/dock…" 4 months ago Up 3 hours 0.0.0.0:5044->5044/tcp, 9600/tcp logstash
7bafa2d70b96 registry:2 "/entrypoint.sh /etc…" 4 months ago Up 4 months 5000/tcp registry
12b0addb3d8c jwilder/nginx-proxy "/app/docker-entrypo…" 9 months ago Up 5 months 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp nginx-proxy
1c79f1d3e839 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/entr…" 9 months ago Up 5 months nginx-proxy-companion
So i think we use this : docker.elastic.co/logstash/logstash-oss:6.1.4
Yes, that's what it looks like. I suppose that image uses the usual jvm.options file to control the JVM configuration, so you should be able to modify that file to your needs and use one of the methods described in https://www.elastic.co/guide/en/logstash/current/docker-config.html to replace the original jvm.options with the one you've changed.
Ok. So, i search for the jvm.options file :
root@docker-01:/# find -type f -name jvm.options
./var/lib/docker/overlay2/6e7b37531e2838d528272508dbe5c176bdf94987370f97c9d9b2780d3f11be31/diff/etc/logstash/jvm.options
./var/lib/docker/overlay2/c6dd5fa31148d1e95eba460cfffb02f595d1afb09675588056a8ec8dd0b2f083/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/99d1be935f1218c9f5492b5f0602357ddc89ffe6710463e4b8ebbce3d272930b/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/822b1aae5f823be2043d73bea807eb882a6fbf0d96fdcf962420128f1c31fd71/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/6652caa224e0df18e9c1a6e5c5953e4970fd4e40ed683dc8d0879462f75ae905/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/7890360de3fbcf9261bf6542498e90077825fccaaec9147f20646b1cfcdfb383/merged/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/7890360de3fbcf9261bf6542498e90077825fccaaec9147f20646b1cfcdfb383/diff/usr/share/logstash/config/jvm.options
And i don't know which one i had to modify...
My logstash pipeline files are here :
/var/lib/logstash/pipeline
Look for the jvm.options inside your Docker container, for example by creating a shell inside it,
docker exec -it logstash bash
and poking around. It's probably in /etc/logstash. You can obtain the contents of the file by running
docker exec logtash cat /etc/logstash/jvm.options > jvm.options
and then you can modify the file and bind-mount it into your container as described in the documentation I linked to.
The jvm.options file is in :
/usr/share/logstash/config/
But it's empty, so i tried to increase the jvm memory with this command :
java -Xms4g -Xmx4g
And i get this message :
OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:
112
Try using the -Djava.io.tmpdir= option to select an alternate temp location.
When i check the available space (inside the logstash container), i get :
bash-4.2$ df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 193379860 187189356 0 100% /
tmpfs 65536 0 65536 0% /dev
tmpfs 6151884 0 6151884 0% /sys/fs/cgroup
/dev/mapper/docker--01--vg-root 193379860 187189356 0 100% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 6151884 0 6151884 0% /proc/scsi
tmpfs 6151884 0 6151884 0% /sys/firmware
and outside :
root@docker-01:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 6131680 0 6131680 0% /dev
tmpfs 1230380 132444 1097936 11% /run
/dev/mapper/docker--01--vg-root 193379860 187259640 0 100% /
tmpfs 6151884 0 6151884 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 6151884 0 6151884 0% /sys/fs/cgroup
/dev/sda1 482922 157228 300760 35% /boot
tmpfs 1230380 0 1230380 0% /run/user/1000
overlay 193379860 187259640 0 100% /var/lib/docker/overlay2/565e17bc395cdd48fcacfb237d097afb8087b20df3e34a2bf8fe09c9e0791a0a/merged
overlay 193379860 187259640 0 100% /var/lib/docker/overlay2/b4e3d8451d08474bc17a8ba50a6b9286d5517d1687b319c6ac6403a7e1e1c236/merged
shm 65536 0 65536 0% /var/lib/docker/containers/12b0addb3d8cfdb84cc2b3e7c1e8d2cd5526c76ba80921c1c66d9648594815ca/shm
shm 65536 0 65536 0% /var/lib/docker/containers/1c79f1d3e8395889364951b5a2d3512b1bd8558a57d1808cb9d19f0122aa3585/shm
overlay 193379860 187259640 0 100% /var/lib/docker/overlay2/4abd7b83a0008cfb8bce2b072d6bea0bfd32058318032184063ce61381339312/merged
shm 65536 0 65536 0% /var/lib/docker/containers/7bafa2d70b96c697a73ede8ebf81357d5c32b909396a4859ccfd955874303b87/shm
tmpfs 1230380 0 1230380 0% /run/user/1002
overlay 193379860 187259640 0 100% /var/lib/docker/overlay2/7890360de3fbcf9261bf6542498e90077825fccaaec9147f20646b1cfcdfb383/merged
shm 65536 0 65536 0% /var/lib/docker/containers/8b377d4515b81281a6f89b990638f10e8d0682b4e8dfdf3244e8c4d2f6be200d/shm
I'm lost between ubuntu, docker, vm.... ! 
I really don't know what to do....
It looks like your /dev/mapper/docker--01--vg-root is full.
is there a solution to increase space ?
I'm sure there is, but how you in your particular case should increase the amount of available space on one of your storage volumes is a question best asked elsewhere.
Thanks a lot for your help Magnus.
Have a nice day.