Unable to Ship logs through Grafana Loki plugin

Elastic Stack Logstash
1

Hello

I am trying to get to work the Grafana loki Plugin for Logstash but I am getting a runtime error, here is the detail.

Grafana Loki plugin was installed successfully by executing the following command:

sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-loki.
Validating logstash-output-loki
Installing logstash-output-loki
Installation successful

However the version that got installed was logstash-output-loki (1.0.3) and not the latest and I think it is due to our current logstash version... Which is 6.4.2, but the installation was successful so it does not represent a problem at this moment for me. (I think that since higher versions throws error during installation)

I added the loki section within the output element from the configuration file and then ran the consistency check for the config prior doing the change.

sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /tmp/testing.loki

Configuration OK
[INFO ] 2022-02-07 21:24:55.114 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

But the services started to crash with the following fatal error within the logs:

[2022-02-05T05:50:19,546][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method append' for #<Array:0x3dad3e98>>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-loki-1.0.3/lib/logstash/outputs/loki/batch.rb:25:in add'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-loki-1.0.3/lib/logstash/outputs/loki.rb:190:in add_entry_to_batch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-loki-1.0.3/lib/logstash/outputs/loki.rb:99:in block in max_batch_size'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-loki-1.0.3/lib/logstash/outputs/loki.rb:98:in block in max_batch_size'", "org/jruby/RubyKernel.java:1292:in loop'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-loki-1.0.3/lib/logstash/outputs/loki.rb:90:in max_batch_size'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-loki-1.0.3/lib/logstash/outputs/loki.rb:86:in `block in register'"]}

I am a bit lost on what the issue could be or the resolution.

Do you have any advice or ideas on how to troubleshoot this issue?

Thanks in advance

2

It is blowing up here, trying to append an entry to an array.

You appear to be running jruby/2.3.0. The .append method as an alias for .push was only added in JRuby 2.5.0 according to this blog.

The gemspec for the plugin does not include a dependency on the jruby version. I am not familiar enough with Ruby development best practices to know whether it should (which would cause the plugin installation to fail).

Your best option would be to upgrade to the latest logstash version, which includes security fixes not in 6.4.2, which is coming up on two years past EOL.

Another option would be to fork the repository for the plugin, edit the .append to be .push, and build the plugin yourself.

3

Hello

Thank you for the prompt response and your contribution.

I just found this: https://github.com/grafana/loki/issues/3349 where the reporter mentioned it has been fixed after the upgrade like you are suggesting. I will definitively consider that and evaluate the risks on going through that path in regards existing input/outputs plugins to AWS S3 buckets. Hopefully there are no conflicts with it.

I'll go through the second option in case the upgrade is not possible from our side.

Thanks again!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.