Correlating information from different indexes

[espogian]

Hi, this is something I have already found in several discussions here and in other places, but I would like to ask for your advice because I'm still not sure about which is the best solution.

I have data coming from a CSV formatted as follows:

timestamp;id;value

And I would like to create Visualizations based on other parameters which are present in another CSV which correlates id with other parameters.
For example, apparently on Kibana I can't configure a stacked-bar visualization based on timestamp and values from one index differentiating the stacks for another parameter which is cointained in another index.

I see from other questions that one solution could be to add this information when parsing the CSV with Logstash using the Elasticsearch plugin.
I'm wondering if there are other (easier) options.

Thank you

[Bill_McConaghy]

Elasticsearch has no notion of joins or something similar. Your best bet is to merge the data at ingest time and put that merged data into an index.

[espogian]

I'm trying with Elasticsearch plugin for Logstash, but I'm receiving this error from Logstash:

[LogStash::Runner] ERROR logstash.plugins.registry - Problems loading a plugin with {:type=>"filter", :name=>"elasticsearch", :path=>"logstash/filters/elasticsearch", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:226:in `namespace_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:162:in `legacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:103:in `plugin'", "(eval):93:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:75:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:165:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:296:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:313:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
[LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'elasticsearch'. Are you sure this is correct? Trying to load the elasticsearch filter plugin resulted in this error: Problems loading the requested plugin named elasticsearch of type filter. Error: NameError NameError"}

I'm running Logstash on a Docker container.
Do you have any idea what's going on?

[espogian]

I think this is a useful link to start: https://www.elastic.co/blog/elasticsearch-docker-plugin-management

[espogian]

I found the solution creating a new container installing the filter plugin logstash-filter-elasticsearch https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

The new Dockerfile can be simple as explained here: https://hub.docker.com/_/logstash/