Try to use aggregate filter

ssasporta · September 11, 2017, 3:06pm

I am trying to use the aggregate filter and that what I am getting in the log:

[2017-09-11T17:53:35,256][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"aggregate", :path=>"logstash/filters/aggregate", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:221:in `namespace_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:157:in `legacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:133:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:175:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:139:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:91:in `plugin'", "(eval):329:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:63:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2

Any Idea?

Where should I set this parameter?

Thanks
Sharon.

magnusbaeck · September 14, 2017, 5:37am

I am trying to use the aggregate filter and that what I am getting in the log:

Is the aggregate filter installed?

Where should I set this parameter?

It's a Logstash command line option.

ssasporta · September 14, 2017, 8:26am

Hi,

No, I didn't know that I need to do special installation for the aggregated filter.

Do you have a link, how to install it and from where?

Thanks
Sharon.

magnusbaeck · September 14, 2017, 8:35am

https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html

Architha · September 20, 2017, 1:56pm

Hi @ssasporta,

You could set workers -w 1 flag either in the logstash command line

logstash -f -w 1 xyz.conf

else in the logstash.yml file under the pipeline settings, by uncommenting the following line:

pipeline.workers: 1

ssasporta · September 28, 2017, 2:06pm

I am getting the following error while trying to install the plugin:

root@vpelastic:[/root]$/usr/share/logstash/bin/logstash-plugin install logstash-filter-aggregate

URI::InvalidURIError: bad URI(is not URI?): 10.232.233.70:8080
                              split at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/common.rb:176
                              parse at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/common.rb:210
                              parse at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/common.rb:747
                                URI at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/common.rb:994
      extract_proxy_values_from_uri at /usr/share/logstash/lib/pluginmanager/proxy_support.rb:47
                    configure_proxy at /usr/share/logstash/lib/pluginmanager/proxy_support.rb:69
                             (root) at /usr/share/logstash/lib/pluginmanager/main.rb:26

Any help?

*Goggle had some ideas regarding the parsing, and for this I need to change the common.rb file.

Thanks
Sharon.

ssasporta · October 1, 2017, 1:17pm

I assume the problem is with the formatting of my http_proxy string.

Still checking how to solve it. In general I assume it should looks like: http://DOMAIN\user:pass@host:port

Will update if this solved my problem.

ssasporta · October 8, 2017, 12:53pm

After been able to install the aggregate filter:

root@vpelastic:[/root]$ /usr/share/logstash/bin/logstash-plugin install --local --no-verify  /home/vpwrk1/filters/logstash-filter-aggregate-2.6.3.gem
start configure_proxy
configure_proxy - proxy value : http://10.232.233.70:8080
http://10.232.233.70:8080
uri=http://10.232.233.70:8080
extract_proxy_values_from_uri input value : http://10.232.233.70:8080
configure_proxy - proxy_settings :
https://10.232.233.70:8080
uri=https://10.232.233.70:8080
extract_proxy_values_from_uri input value : https://10.232.233.70:8080
uri=/home/vpwrk1/filters/logstash-filter-aggregate-2.6.3.gem
uri=https://artifacts.elastic.co/downloads/logstash-plugins//home/vpwrk1/filters/logstash-filter-aggregate-2.6.3.gem//home/vpwrk1/filters/logstash-filter-aggregate-2.6.3.gem-5.4.1.zip
Installing logstash-filter-aggregate
uri=https://rubygems.org
uri=https://rubygems.org

Installation successful

I am now trying to use it, but getting error during Logstash initialization:

[2017-10-08T14:05:52,784][DEBUG][logstash.plugins.registry] Problems loading the plugin with {:type=>"filter", :name=>"aggregate"}
[2017-10-08T14:05:52,786][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'aggregate'. Are you sure this is correct? Trying to load the aggregate filter plugin resulted in this error: Problems loading the requested plugin named aggregate of type filter. Error: NameError NameError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:175:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:139:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:91:in `plugin'", "(eval):3883:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:63:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

Any idea?

Architha · October 9, 2017, 3:02pm

Hi @ssasporta,

Here's another way to install the Aggregate Plugin on Windows -

Download the logstash-filter-aggregate plugin from the this link. Unzip and save.
Open the gemfile in the Logstash folder and edit the file by adding the following text on the last line: gem "logstash-filter-aggregate", :path =>"path-to-aggregate-filter/logstash-filter-aggregate"
Keep in mind that the path to the aggregate plugin folder has to be given with forward-slash "/"

Run Logstash ! you are good to go.

ssasporta · October 10, 2017, 11:55am

Hi,

Thanks for your help.

I am working on Linux Redhat so I assume it isn't relevant.

Thanks,
Sharon.

system · November 7, 2017, 11:55am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.