Problem installing logstash-filter-cipher

Elastic Stack Logstash
1

Hi all,

I'm trying to install chiper filter but unfortunately I'm gettin this error:

bin/logstash-plugin install logstash-filter-cipher
Validating logstash-filter-cipher
Installing logstash-filter-cipher
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core-plugin-api":
  In snapshot (Gemfile.lock):
    logstash-core-plugin-api (= 2.1.12)

  In Gemfile:
    logstash-devutils (>= 0) java depends on
      logstash-core-plugin-api (<= 2.99, >= 2.0) java

    logstash-input-s3 (>= 0) java depends on
      logstash-mixin-aws (>= 0) java depends on
        logstash-core-plugin-api (<= 2.99, >= 1.60) java

the error continues till it says

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Bundler could not find compatible versions for gem "logstash":
  In Gemfile:
    logstash-filter-cipher (>= 0) java depends on
      logstash (< 2.0.0, >= 1.4.0) java
Could not find gem 'logstash (< 2.0.0, >= 1.4.0) java', which is required by gem 'logstash-filter-cipher (>= 0) java', in any of the sources.

I don't know if chiper filter is not supported yet, I'm using logstash 5.2.

Suggestions?

Thanks a lot

2

Sorry, somebody can help here?

3

This filter isn't Logstash 5.x-compatible yet.

4

I don't know what happened.

Today I've installed Logstash 5.2.1 again, and successfully installed the cipher-filter!!!

The problem now is when i test the filter, i get this error

[2017-02-28T11:33:43,245][WARN ][logstash.filters.cipher ] Exception catch on cipher filter {:event=>2017-02-28T10:33:43.237Z DonVito-A-tono prov, :error=>#<NoMethodError: Direct event field references (i.e. event['field']) have been disabled in favor of using event get and set methods (e.g. event.get('field')). Please consult the Logstash 5.0 breaking changes documentation for more details.>}

using this pipeline

input{
	file{
		path => "/home/vittorio/Documents/dpapi.json"
		codec => "json"
	}
}

filter{
cipher {
			algorithm => "aes-256-cbc"
			cipher_padding => 1

			# Use a static "iv"
			#iv = "1234567890123456"

			# OR use a random IV per encryption
			iv_random_length => 16

			key => "12345678901234567890123456789012"
			key_size => 32

			mode => "encrypt"
			remove_field => ["message"]

	}

}

output {stdout { codec => rubydebug }}

in the file this is what i have

{"class": "DPAPIINTERNAL","ts": "1:3075","aw": "sdsEH", "message":"prov"}

5

I'm surprised that you were able to install the plugin, because it isn't 5.0-compatible and the error message you get points to that problem.

6

I cloned the logstash-filter-cipher, adding newest gem dependencies. That is why I was able to install it.

I'm not finding a good solution to make the chiper filter work by myself.
Are there any other methods that might help me encrypt data directly from logstash?

7

I cloned the logstash-filter-chiper, adding newest gem dependencies. That is why I was able to install it.

Oh. Including that in your previous post would've cleared things up.

I'm not finding a good solution to make the chiper filter work by myself.

Previously fields were both read and written to via event['fieldname'] but as of Logstash 5.0 you need to use event.get('fieldname') for reading and event.set('fieldname', 'desired value') for writing.

See also Event API | Logstash Reference [8.11] | Elastic.

8

So ideally i should use

 chiper{
    algorithm => ....
    mode => "encrypt"
    remove_field => ["event.get('message')"]
}

to make it work? or the new logstash doesn't have this previous syntax enabled?

9

No. You need to modify the cipher plugin's source code and make the replacements I described.

10

I wish...

Never coded using Ruby and also I'm not a good developer..

I tried to modify the chiper.rb file, each time i saw

if (event[@source].nil? || event[@source].empty?)

i changed with if (event.get(@source).nil? || event.get(@source).empty?)

after that, when executing the pipeline many exceptions were lanuched :smiley:

11

filter modified successfully ! thanks for the suggestions @magnusbaeck

12

Hi everyone,

@magnusbaeck I've replaced the source code as you described and everything works fine when as input I have, for example, simple files.

Once i use rabbitmq I'm having problems, for instance the pipeline gets crazy...

[2017-03-03T10:31:30,526][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<OpenSSL::Cipher::CipherError: key not specified>, "backtrace"=>["org/jruby/ext/openssl/Cipher.java:933:in `reset'", "/home/vittorio/logstash-5.2.1/vendor/local_gems/3b3689f9/logstash-filter-cipher-2.0.5/lib/logstash/filters/cipher.rb:217:in `init_cipher'", "/home/vittorio/logstash-5.2.1/vendor/local_gems/3b3689f9/logstash-filter-cipher-2.0.5/lib/logstash/filters/cipher.rb:208:in `filter'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filter_delegator.rb:41:in `multi_filter'", "(eval):98:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):94:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):75:in `filter_func'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:300:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:201:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:200:in `each'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:299:in `filter_batch'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:287:in `worker_loop'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:263:in `start_workers'"]}
[2017-03-03T10:31:30,731][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<OpenSSL::Cipher::CipherError: key not specified>, :backtrace=>["org/jruby/ext/openssl/Cipher.java:933:in `reset'", "/home/vittorio/logstash-5.2.1/vendor/local_gems/3b3689f9/logstash-filter-cipher-2.0.5/lib/logstash/filters/cipher.rb:217:in `init_cipher'", "/home/vittorio/logstash-5.2.1/vendor/local_gems/3b3689f9/logstash-filter-cipher-2.0.5/lib/logstash/filters/cipher.rb:208:in `filter'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/filter_delegator.rb:41:in `multi_filter'", "(eval):98:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):94:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):75:in `filter_func'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:300:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:201:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:200:in `each'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:299:in `filter_batch'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:287:in `worker_loop'", "/home/vittorio/logstash-5.2.1/logstash-core/lib/logstash/pipeline.rb:263:in `start_workers'"]}

the key not specified is a mistery because i've specified it..
I tried with a lower prefetch_count => 100 but at some point, it shows up the exception..
It looks like something is wrong with the filter, maybe it doesn't support when rabbitmq has lots of messsages in the queue...

Thanks

13

the problem still there.... any suggestions?
thanks

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.