ETL tool for ElasticSearch

Elastic Stack Elasticsearch
1

Hi Guys,
do you know any ETL tool where is possible to have a connector to ElasticSearch and get all data that i need?
I was trying talend studio (big data) and oracle data integrator, but without results.
if with this tools i cannot find a way to extract the data, Do you think that is better extract data and write it to a file and after use the ETL tool to load data from file?

thank you

2

If I read the question correctly, you want to copy documents from Elasticsearch to another system. Correct?

Elastic has the Logstash product and there is an Elasticsearch input that can process the results of a ES query.

What is the destination of the docs?

3

Are you trying to Move documents from ES to another DB or are you trying to move documents into ES?

Please clarify.

Thanks!

4

I have data in elasticsearch and i want to create dashboards in powerBI with some data saved on ElasticSearch. So, my intention is create a ETL process that load data from ES, treat the data and insert the new data on Data Warehouse (im using MySQL).
My problem, is that i didnt find any connector for elasticsearch, on Talend Big Data Studion (free edition) neither on ODI oracle.
Do i need to export the data to files and after use the ETL tool or do you know some other way to do that?

5

Why not use Kibana?

1 Like
6

I'm curious to learn what PowerBI can do for you that Kibana can't :wink:

Have you tried Dremio yet?

1 Like
7

Thank you, i didn't try yet.
PowerBI you can create more "professional" dashboards and create reports for customers. I use kibana but just inside team.

8

Well, only until Canvas comes out, that is :wink:

And to be honest, Kibana 6.1 with dashboard-only role, dashboard input controls and more leaves nothing to be desired

1 Like
9

I want import data from ES and my output would be mysql. Can be possible do that with logstash?

10

Yes. logstash would work perfectly for this use case - you can read the data from elasticsearch, apply any transformations needed and then load to mysql directly or output to a file and load from that.

11

Perfect, i will try this :). thank you

12

An alternative approach, if you don't want to use Logstash, you can use Pentaho with a few lines of simple custom python code that you could customize to do what you want.

13

thank you for your answer.
Im trying logstash but i'm getting some errors to add jdbc output plugin...

14

It'd be worth creating a new thread in #logstash with your problems so we can help :slight_smile:

15

The CData Elasticsearch Drivers (https://www.cdata.com/drivers/elasticsearch/) should be able to plug-in to just about any ETL tool and help you accomplish what you are trying to do. The driver interfaces are supported by every modern data movement application.

16

@ericmad please be clear that you are an employee of this organisation and you are promoting a paid product as an employee.

2 Likes
17

You might wanna take a look at Apache NiFi which has processors (connectors) for a ton of stuff, including Elasticsearch and SQL.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.