Installing Gem for Ruby Filter Plugin

Elastic Stack Logstash
1

Hello there,

I am trying to install the jdbc-mysql gem in a dockerized Logstash (after I miserably failed to set Xcext.enabled=true and use the mysql2 gem).

In the Dockerfile I am first running this

RUN env GEM_HOME=/opt/logstash/vendor/bundle/jruby/1.9 /usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/vendor/jruby/bin/gem install jdbc-mysql

to install the gem and then the following to add it to the filter's gemspec:

RUN head -n -2 ./vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-3.0.2/logstash-filter-ruby.gemspec > ./vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-3.0.2/logstash-filter-ruby.gemspec.temp; mv ./vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-3.0.2/logstash-filter-ruby.gemspec.temp ./vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-3.0.2/logstash-filter-ruby.gemspec
RUN echo "s.add_runtime_dependency 'jdbc-mysql\nend'" >> ./vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-3.0.2/logstash-filter-ruby.gemspec

But it's still not working, but complainig like so:

ERROR logstash.filters.ruby - Ruby exception occurred: no such file to load -- jdbc/mysql

Is this actually the right way to do it? Or is there another, better and easier way to achieve this? All I really want to do is query MySQL from that filter. :confused:

How to add new gem for my plugin usage? seems to imply this is the way to go, whereas Logstash not finding installed gems didn't even get an answer...

I am lost ...help!

1 Like
2

By the way, I got the idea on how to do this from here: https://stackoverflow.com/questions/25984620/how-to-add-gems-into-logstash, as I couldn't find any official documentation.

3

Anyone?

4

bump

I am horribly stuck here...

5

At this point I am wondering: What does it mean that my question is drowning in others which all get replies? Is it silly? Is it stupid? Is it hard?

In case of the former, please educate me!

In case of the latter, and in the case that you don't feel like you can help me, maybe you know someone who could? If that is the case, I'd appreciate, if you could call that person's attention to this thread.

Cheers
Will

1 Like
6

Still stuck...

1 Like
7

Should I provide more information?

1 Like
8

https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_streaming.html

Neat, looks like I am one lucky guy! I will try that later!

On the other hand I would still like to find out how to add gems though... surely someone around here must know.

1 Like
9

I am a monkey.

While trying out different things, I must have at some point taken out the Dockerfile command to append the gem to the Gemfile. It's working now. Always. Investigate. Exception. Message.

An ashamed but happy monkey!

So, for completeness' sake and future generations:
RUN echo 'gem "jdbc-mysql"' >> Gemfile
... is what was missing.

2 Likes
10

Let me know how you get on with the jdbc_streaming. (I just wrote it, but I don't have a real personal use case and would appreciate feedback - especially regarding performance or memory consumption of the cache)

11

Hey @guyboertje, I did end up using the gem and connecting to the MySQL from within the Ruby filter after all. But should I start using the jdbc_streaming at some point, I'll be more than happy to let you know how I do.

2 Likes
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.