Hello there,
All the sudden, our developers reported that logstash started working abnormally.
I took a look on the all nodes of ELK, kibana and elastisticsearch nodes are working fine.
I'm able to log on to kibana and see that beats are sending data to Elastic via logstash.
However, there is not log entries in the logstash.log and in the /var/log/messages, I can see that the logstash keeps starting and stopping, the following items in the /var/log/messages logs keep coming:
....................
Oct 29 16:20:22 logstashhost systemd: Started logstash.
Oct 29 16:20:22 logstashhost systemd: Starting logstash...
Oct 29 16:20:25 logstashhost logstash: [ERROR] 2018-10-29 16:20:25.758 [main] Logstash - java.lang.ExceptionInInitializerError
Oct 29 16:20:25 logstashhost systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
Oct 29 16:20:25 logstashhost systemd: Unit logstash.service entered failed state.
Oct 29 16:20:25 logstashhost systemd: logstash.service failed.
Oct 29 16:20:26 logstashhost systemd: logstash.service holdoff time over, scheduling restart.
Oct 29 16:20:26 logstashhost systemd: Started logstash.
Oct 29 16:20:26 logstashhost systemd: Starting logstash.
......
Nothing changed on logstash.yml, neither on the pipeline.conf files..
No entries in the logstash log file. But I do see the filebeat is sending data to kibana in almost real time.
But the pipelines to fetch data from AWS s3 buckets stopped working, we used logstash s3 input plugins to fetch data from s3 buckets to logstash and then to elasticsearch, the plugins stopped working. Again, nothing changed on the pipeline conf on those s3 plugins...
The only change I can see on this linux is that the system admin upgrade java from 1.8.0_172 to 1.8.0_181 on the logstash server... but on the elasticsearch nodes, the java is still at 1.8.0_172.
Does this java upgrade cause the problem?
here is the Linus system information:
Linux version 3.10.0-862.14.4.el7.x86_64 (mockbuild@x86-040.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Fri Sep 21 09:07:21 UTC 2018
Java now on the logstash server is:
[root@logstashhost logstash]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
Before upgrade, it was:
Java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
Again, nothing changed on the logstash.yml and all the pipeline.conf file....
Filebeat still sends data from remote servers to logstash then to elasticsearch, I can see the beats data comes in on kibana almost realtime, but the s3 input plugins (the s3 pipelines) stopped working.
Could you please please take a look and let us know how to fix this?
Thank you in advance
Li