Logstash crashes when trying to install cloudwatch_logs plugin

Elastic Stack Elasticsearch
1

Logstash version: 8.15.1
OS: Amazon Linux 2
All packages updated

Steps to reproduce:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

in /etc/yum.repos.d/logstash.repo

[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install -y logstash

in /etc/logstash/logstash.yml, all settings as they are except:

api.http.host: MY_IP

in /etc/logstash/jvm.options, all settings as they are except I modified

-Xms3762m
-Xmx3762m

in /etc/logstash/conf.d/ I've added my-input.conf:

input {
     azure_event_hubs {....}
}

in /etc/logstash/conf.d/ I've added my-output.conf:

output {
     rabbitmq {...}
}

/usr/share/logstash/bin/logstash-plugin install logstash-input-azure_event_hubs

Installed successfully

Started logstash -> I got an error related to the azure plugin, but it is not the object of this topic. Logstash started.

/usr/share/logstash/bin/logstash-plugin install logstash-input-cloudwatch_logs

Output:

/usr/share/logstash/bin/logstash-plugin install logstash-input-cloudwatch_logs
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-input-cloudwatch_logs
Resolving mixin dependencies
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      date (>= 0)
      Available/installed versions of this gem:
      - 3.3.4
      - 3.3.3
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
Installing logstash-input-cloudwatch_logs

and it remains like this unless I interrupt it.

After that, when I try to restart logstash the output is:

Sep 12 21:56:05 ip-10-7-188-140 systemd: Started logstash.
Sep 12 21:56:05 ip-10-7-188-140 logstash: Using bundled JDK: /usr/share/logstash/jdk
Sep 12 21:56:15 ip-10-7-188-140 logstash: [FATAL] 2024-09-12 21:56:15.253 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
Sep 12 21:56:15 ip-10-7-188-140 logstash: If you are a user, this is a bug.
Sep 12 21:56:15 ip-10-7-188-140 logstash: If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set.
Sep 12 21:56:15 ip-10-7-188-140 systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 12 21:56:15 ip-10-7-188-140 systemd: Unit logstash.service entered failed state.
Sep 12 21:56:15 ip-10-7-188-140 systemd: logstash.service failed.
Sep 12 21:56:15 ip-10-7-188-140 systemd: logstash.service holdoff time over, scheduling restart.
Sep 12 21:56:15 ip-10-7-188-140 systemd: Stopped logstash.
Sep 12 21:56:15 ip-10-7-188-140 systemd: Started logstash.
Sep 12 21:56:15 ip-10-7-188-140 logstash: Using bundled JDK: /usr/share/logstash/jdk
Sep 12 21:56:23 ip-10-7-188-140 systemd: Stopping logstash...
Sep 12 21:56:23 ip-10-7-188-140 systemd: logstash.service: main process exited, code=exited, status=143/n/a
Sep 12 21:56:23 ip-10-7-188-140 systemd: Stopped logstash.
Sep 12 21:56:23 ip-10-7-188-140 systemd: Unit logstash.service entered failed state.
Sep 12 21:56:23 ip-10-7-188-140 systemd: logstash.service failed.

Even trying to get the version of logstash resulted in:

/usr/share/logstash/bin/logstash version
Using bundled JDK: /usr/share/logstash/jdk
[FATAL] 2024-09-12 21:59:52.938 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
If you are a user, this is a bug.
If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set.

While reproducing this, you can omit the azure part. Just trying to install that plugin makes logstash unusable.

I've noticed the same behaviour if instead of trying to install cloudwatch_logs plugin I try an upgrade of the azure plugin

/usr/share/logstash/bin/logstash-plugin update logstash-input-azure_event_hubs

Am I doing something wrong? Thank you for any suggestion.

2

The logstash-input-cloudwatch_logs is made by a third party, not by Elastic, it was discontinued by the developer last year since it hasn't been updated since 2018.

There is no guarantee that it will work with newer Logstash versions and no fix as this is not mantained anymore.

To get logs from Cloudwatch it is recommended to use Elastic Agent now.

3

Thank you for your reply Leandro. However, is it normal that the service crashes and I cannot restart it even if I uninstall and reinstall it?

4

Also, the same happens when I update the azure plugin which is supported by Elastic.

5

I would not expected that, but maybe something broke during this process.

Have completely uninstalled and reinstall Logstash or just the plugin?

The azure event hub is bundled in Logstash, there is no need to install it.

I would recommend that you completely uninstall Logstash and try to install it again.

6

I've removed logstash and all related files, reinstalled it and it started successfully.
However, I still believe that this is a bug.
On a completely new machine I tried this:

  • installed logstash and added some basic config files. The service started successfully.
    Then I did this:
/usr/share/logstash/bin/logstash-plugin install logstash-input-cloudwatch_logs
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-input-cloudwatch_logs
Resolving mixin dependencies
Installing logstash-input-cloudwatch_logs



^C
/usr/share/logstash/bin/logstash-plugin install logstash-input-github
Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-input-github
Resolving mixin dependencies
WARN: Unresolved or ambiguous specs during Gem::Specification.reset:
      date (>= 0)
      Available/installed versions of this gem:
      - 3.3.4
      - 3.3.3
WARN: Clearing out unresolved specs. Try 'gem cleanup <gem>'
Please report a bug if this causes problems.
^C
service logstash start

The output was the one posted initially here:

Sep 13 05:34:41 MY_IP systemd: Started logstash.
Sep 13 05:34:41 MY_IP logstash: Using bundled JDK: /usr/share/logstash/jdk
Sep 13 05:34:50 MY_IP logstash: [FATAL] 2024-09-13 05:34:50.984 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
Sep 13 05:34:50 MY_IP logstash: If you are a user, this is a bug.
Sep 13 05:34:50 MY_IP logstash: If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set.
Sep 13 05:34:51 MY_IP systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
Sep 13 05:34:51 MY_IP systemd: Unit logstash.service entered failed state.
Sep 13 05:34:51 MY_IP systemd: logstash.service failed.
Sep 13 05:34:51 MY_IP dhclient[2238]: XMT: Solicit on eth0, interval 128720ms.
Sep 13 05:34:51 MY_IP systemd: logstash.service holdoff time over, scheduling restart.
Sep 13 05:34:51 MY_IP systemd: Stopped logstash.

I believe that logstash-input-github is a plugin that is not bundled and should have installed successfully even if logstash-input-cloudwatch_logs didn't, right?
In this case the solution was to again remove all logstash related files and reinstall it.
Thank you once more.