Logstash exceptionCaught() event

Khaled_Saidi · July 26, 2018, 3:39pm

Hi,

I use filebeat to send logs to logstash who is running in a docker's container on ubuntu.
Since the gathered log files format was modified, i have a lot of grokparse failure. I don't know if there is a relation but know, my logstash server crashes.
I have this error :

Unhandled Java exception: java.lang.BootstrapMethodError: call site initialization exception
java.lang.BootstrapMethodError: call site initialization exception
[.....]
Caused by:
[2018-07-26T14:08:42,862][WARN ][io.netty.channel.AbstractChannelHandlerContext] The exceptionCaught() event that was failed to submit was:
java.util.concurrent.RejectedExecutionException: event executor terminated
[...]
[2018-07-26T14:08:45,241][WARN ][io.netty.channel.AbstractChannelHandlerContext] Failed to submit an exceptionCaught() event.
java.util.concurrent.RejectedExecutionException: event executor terminated
[...]
[2018-07-26T15:40:15,319][WARN ][logstash.filters.json    ] Error parsing json {:source=>"json_column", :raw=>"base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5[...]AX9KtJ4U09Rwv6VMnhewH8P6VctdBs4Xyq/pWkthCBgCl+ww5zinCxh9KclnEvIWphAg7UxrZG6037HF6Uv2SL0pPscXpSGyi9KQ2MJ7Uz+zYPSl/s6D+7SjT4B/DThYQf3amS2iRcBRX//2Q==\"}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'base64': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5[...]AX9KtJ4U09Rwv6VMnhewH8P6VctdBs4Xyq/pWkthCBgCl+ww5zinCxh9KclnEvIWphAg7UxrZG6037HF6Uv2SL0pPscXpSGyi9KQ2MJ7Uz+zYPSl/s6D+7SjT4B/DThYQf3amS2iRcBRX//2Q=="}"; line: 1, column: 8]>}
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Unable to create java_pid1.hprof: File exists

here are my logstash container stats :

CONTAINER ID        NAME          CPU %        MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
8b377d4515b8        logstash      78.27%      1.279GiB / 11.73GiB   10.90%              9.57MB / 22.5MB     19.6MB / 5.2MB      78

Have you an idea on how to fix this ?

Thx a lot for your help.

BR,
Khaled

magnusbaeck · August 4, 2018, 7:29pm

The JVM ran out of memory. Have you tried increasing the JVM heap?

Khaled_Saidi · August 6, 2018, 9:42am

Thank you Magnus.
I'm running logstash in a docker container on an ubuntu distribution, and i'm really not comfortable with this
So, i would be glad if you give me the command line to increase the JVM heap memory through the docker container.

Thanks à lot for your help.

magnusbaeck · August 6, 2018, 10:10am

Which Docker image are you using?

Khaled_Saidi · August 6, 2018, 12:30pm

i don't know which image was used. here is the output for a docker ps -a command :

CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS                   PORTS                                      NAMES
7fa5af7743fe        java                                            "java -Xmx4096m"         3 hours ago         Exited (1) 3 hours ago                                              zen_payne
be7856979552        java                                            "-Xmx4096m"              3 hours ago         Created                                                             suspicious_colden
9c154454b4e9        java                                            "-Xmx4096m"              3 hours ago         Created                                                             loving_lumiere
d6384d68cbb8        java                                            "java -Xmx4096m"         3 hours ago         Exited (1) 3 hours ago                                              elastic_lamport
c01fd3e78163        java                                            "java -XshowSettings…"   3 hours ago         Exited (0) 3 hours ago                                              musing_blackwell
974b8336261d        java                                            "java -XshowSettings…"   3 hours ago         Exited (1) 3 hours ago                                              goofy_austin
c19469e87258        java                                            "java -Xmx10000m -X2…"   3 hours ago         Exited (1) 3 hours ago                                              determined_villani
c526296033af        java                                            "-Xmx10000m -X2000m …"   3 hours ago         Created                                                             zealous_shaw
9590a1890b17        java                                            "java -XshowSettings…"   3 hours ago         Exited (0) 3 hours ago                                              festive_ptolemy
05d2f68696cc        logstash                                        "/docker-entrypoint.…"   10 days ago         Exited (0) 10 days ago                                              relaxed_agnesi
8b377d4515b8        docker.elastic.co/logstash/logstash-oss:6.1.4   "/usr/local/bin/dock…"   4 months ago        Up 3 hours               0.0.0.0:5044->5044/tcp, 9600/tcp           logstash
7bafa2d70b96        registry:2                                      "/entrypoint.sh /etc…"   4 months ago        Up 4 months              5000/tcp                                   registry
12b0addb3d8c        jwilder/nginx-proxy                             "/app/docker-entrypo…"   9 months ago        Up 5 months              0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nginx-proxy
1c79f1d3e839        jrcs/letsencrypt-nginx-proxy-companion          "/bin/bash /app/entr…"   9 months ago        Up 5 months                                                         nginx-proxy-companion

So i think we use this : docker.elastic.co/logstash/logstash-oss:6.1.4

magnusbaeck · August 6, 2018, 1:17pm

Yes, that's what it looks like. I suppose that image uses the usual jvm.options file to control the JVM configuration, so you should be able to modify that file to your needs and use one of the methods described in https://www.elastic.co/guide/en/logstash/current/docker-config.html to replace the original jvm.options with the one you've changed.

Khaled_Saidi · August 6, 2018, 2:38pm

Ok. So, i search for the jvm.options file :

root@docker-01:/# find -type f -name jvm.options
./var/lib/docker/overlay2/6e7b37531e2838d528272508dbe5c176bdf94987370f97c9d9b2780d3f11be31/diff/etc/logstash/jvm.options
./var/lib/docker/overlay2/c6dd5fa31148d1e95eba460cfffb02f595d1afb09675588056a8ec8dd0b2f083/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/99d1be935f1218c9f5492b5f0602357ddc89ffe6710463e4b8ebbce3d272930b/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/822b1aae5f823be2043d73bea807eb882a6fbf0d96fdcf962420128f1c31fd71/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/6652caa224e0df18e9c1a6e5c5953e4970fd4e40ed683dc8d0879462f75ae905/diff/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/7890360de3fbcf9261bf6542498e90077825fccaaec9147f20646b1cfcdfb383/merged/usr/share/logstash/config/jvm.options
./var/lib/docker/overlay2/7890360de3fbcf9261bf6542498e90077825fccaaec9147f20646b1cfcdfb383/diff/usr/share/logstash/config/jvm.options

And i don't know which one i had to modify...
My logstash pipeline files are here :

/var/lib/logstash/pipeline

magnusbaeck · August 6, 2018, 5:51pm

Look for the jvm.options inside your Docker container, for example by creating a shell inside it,

docker exec -it logstash bash

and poking around. It's probably in /etc/logstash. You can obtain the contents of the file by running

docker exec logtash cat /etc/logstash/jvm.options > jvm.options

and then you can modify the file and bind-mount it into your container as described in the documentation I linked to.

Khaled_Saidi · August 7, 2018, 9:48am

The jvm.options file is in :

/usr/share/logstash/config/

But it's empty, so i tried to increase the jvm memory with this command :

java -Xms4g -Xmx4g

And i get this message :

OpenJDK 64-Bit Server VM warning: Insufficient space for shared memory file:
   112
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

When i check the available space (inside the logstash container), i get :

bash-4.2$ df
Filesystem                      1K-blocks      Used Available Use% Mounted on
overlay                         193379860 187189356         0 100% /
tmpfs                               65536         0     65536   0% /dev
tmpfs                             6151884         0   6151884   0% /sys/fs/cgroup
/dev/mapper/docker--01--vg-root 193379860 187189356         0 100% /etc/hosts
shm                                 65536         0     65536   0% /dev/shm
tmpfs                             6151884         0   6151884   0% /proc/scsi
tmpfs                             6151884         0   6151884   0% /sys/firmware

and outside :

root@docker-01:~# df
Filesystem                      1K-blocks      Used Available Use% Mounted on
udev                              6131680         0   6131680   0% /dev
tmpfs                             1230380    132444   1097936  11% /run
/dev/mapper/docker--01--vg-root 193379860 187259640         0 100% /
tmpfs                             6151884         0   6151884   0% /dev/shm
tmpfs                                5120         0      5120   0% /run/lock
tmpfs                             6151884         0   6151884   0% /sys/fs/cgroup
/dev/sda1                          482922    157228    300760  35% /boot
tmpfs                             1230380         0   1230380   0% /run/user/1000
overlay                         193379860 187259640         0 100% /var/lib/docker/overlay2/565e17bc395cdd48fcacfb237d097afb8087b20df3e34a2bf8fe09c9e0791a0a/merged
overlay                         193379860 187259640         0 100% /var/lib/docker/overlay2/b4e3d8451d08474bc17a8ba50a6b9286d5517d1687b319c6ac6403a7e1e1c236/merged
shm                                 65536         0     65536   0% /var/lib/docker/containers/12b0addb3d8cfdb84cc2b3e7c1e8d2cd5526c76ba80921c1c66d9648594815ca/shm
shm                                 65536         0     65536   0% /var/lib/docker/containers/1c79f1d3e8395889364951b5a2d3512b1bd8558a57d1808cb9d19f0122aa3585/shm
overlay                         193379860 187259640         0 100% /var/lib/docker/overlay2/4abd7b83a0008cfb8bce2b072d6bea0bfd32058318032184063ce61381339312/merged
shm                                 65536         0     65536   0% /var/lib/docker/containers/7bafa2d70b96c697a73ede8ebf81357d5c32b909396a4859ccfd955874303b87/shm
tmpfs                             1230380         0   1230380   0% /run/user/1002
overlay                         193379860 187259640         0 100% /var/lib/docker/overlay2/7890360de3fbcf9261bf6542498e90077825fccaaec9147f20646b1cfcdfb383/merged
shm                                 65536         0     65536   0% /var/lib/docker/containers/8b377d4515b81281a6f89b990638f10e8d0682b4e8dfdf3244e8c4d2f6be200d/shm

I'm lost between ubuntu, docker, vm.... !
I really don't know what to do....

magnusbaeck · August 7, 2018, 9:56am

It looks like your /dev/mapper/docker--01--vg-root is full.

Khaled_Saidi · August 7, 2018, 10:23am

is there a solution to increase space ?

magnusbaeck · August 7, 2018, 11:07am

I'm sure there is, but how you in your particular case should increase the amount of available space on one of your storage volumes is a question best asked elsewhere.

Khaled_Saidi · August 7, 2018, 12:18pm

Thanks a lot for your help Magnus.
Have a nice day.

system · September 4, 2018, 12:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.