Logstash vs rivers for reading data from SQL Server

Elastic Stack Elasticsearch
1

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/. Can
someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1173b62c-afd3-4d2d-9a3f-ba423ed7ede4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey,

maybe you should ask your developers, why they recommended logstash for
this, I cant follow here (perhaps there is some export functionality in
your SQL server, which an input of logstash can use?). Would be interested
in reasons in this case.

--Alex

On Wed, Jan 8, 2014 at 5:26 PM, jsp jayasunderam@gmail.com wrote:

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/.
Can someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1173b62c-afd3-4d2d-9a3f-ba423ed7ede4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM-X070J9GKmx7ysce1KCnX5yinMoeDhkkwHfDV%3D_BqDwA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Currently logstash provides an api to get data through sqlite. And have
been looking at how it is being used for sqlite and will try to replicate
that for Sql Server.
Will post code if I get it working.
Thanks
--J

On Thursday, 9 January 2014 00:23:36 UTC-8, Alexander Reelsen wrote:

Hey,

maybe you should ask your developers, why they recommended logstash for
this, I cant follow here (perhaps there is some export functionality in
your SQL server, which an input of logstash can use?). Would be interested
in reasons in this case.

--Alex

On Wed, Jan 8, 2014 at 5:26 PM, jsp <jayasu...@gmail.com <javascript:>>wrote:

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/.
Can someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1173b62c-afd3-4d2d-9a3f-ba423ed7ede4%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9a0f36d8-1868-4608-9c05-558bff9b48ac%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Consider the fact that rivers could be deprecated in future versions of
elasticsearch:

On Wednesday, January 8, 2014 4:26:58 PM UTC, jsp wrote:

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/.
Can someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/136bb9bf-3056-464a-bd37-fb3c5674da41%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks for the link and heads-up, Karol.

Brian

On Friday, January 10, 2014 4:55:29 PM UTC-5, Karol Gwaj wrote:

Consider the fact that rivers could be deprecated in future versions of
elasticsearch:

Sign Up | LinkedIn

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0239cbc4-01cb-432c-a5d9-fc0b95279bff%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Thanks Karl,
Do you have any recommendation for alternatives to using rivers. i want to
read data from sql db/tables.
thanks
jay

On Friday, 10 January 2014 13:55:29 UTC-8, Karol Gwaj wrote:

Consider the fact that rivers could be deprecated in future versions of
elasticsearch:

Sign Up | LinkedIn

On Wednesday, January 8, 2014 4:26:58 PM UTC, jsp wrote:

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/.
Can someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/890a3acc-74f9-4999-b42d-790c78a97c39%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

IMHO, some options:

  • ETL tool like Talend
  • a batch process you need to write
  • modify application which create those data and push to Elasticsearch from it

I prefer the latter as your search engine will be well synchronized with your database.
Also, it will be a gain, database performance wise, as you don't need to read again your DB to inject into ES.

My 2 cents

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 janv. 2014 à 04:12, jsp jayasunderam@gmail.com a écrit :

Thanks Karl,
Do you have any recommendation for alternatives to using rivers. i want to read data from sql db/tables.
thanks
jay

On Friday, 10 January 2014 13:55:29 UTC-8, Karol Gwaj wrote:
Consider the fact that rivers could be deprecated in future versions of elasticsearch:
Sign Up | LinkedIn

On Wednesday, January 8, 2014 4:26:58 PM UTC, jsp wrote:
Hi,
I am looking at implementing ES to index & query data that I get from my SQL Server databases/tables. I was initially using river to read data from Sql server tables but one of the developers in my team recommended looking at using logstash. Can anyone comment to any benefits of using one over another? I have not been able to find any documentation regarding reading data from SQL server using logstash. Can someone point me to a guide on how to get started with logstash & sql server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/890a3acc-74f9-4999-b42d-790c78a97c39%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E23A0F29-2BE9-4948-9082-1817087F0FAE%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

8

i agree with David suggestions,

here is link to topic that is also touching river deprecation (with
suggested replacement(s)):
https://groups.google.com/d/msg/elasticsearch/-nZQheK0J-g/1xhVm9Y5_dgJ

BTW, if you really feel like river is the best solution for you, then you
should go for it
even if elasticsearch team will deprecate rivers from the core
implementation, there will be some community contributed solution to
replace it
i think the amount of elasticsearch applications using rivers is too big to
get rid of them completely

On Saturday, January 11, 2014 3:12:53 AM UTC, jsp wrote:

Thanks Karl,
Do you have any recommendation for alternatives to using rivers. i want
to read data from sql db/tables.
thanks
jay

On Friday, 10 January 2014 13:55:29 UTC-8, Karol Gwaj wrote:

Consider the fact that rivers could be deprecated in future versions of
elasticsearch:

Sign Up | LinkedIn

On Wednesday, January 8, 2014 4:26:58 PM UTC, jsp wrote:

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/.
Can someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f613f7f0-4c69-4e9f-bd2f-9b2c1a3c5f0e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

I just started a gathering framework plugin for fetching and indexing data
to Elasticsearch, with scalable components. It's not complete yet of
course. Next, I intend to create an alternative to the JDBC river, and if
that works out well, for other kinds of rivers too :slight_smile:

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFp7QezAWbw0144BS21GeDXO-0f90qbj-rYwAQ9FMnuaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

This is a little late but.

I agree with the third option, if you can. But you should publish to a
queue and then that goes to ES. That way you can use a 2 phase commit to
the DB and to the queue.

Even with option 1 and 2 ... use a queue. Talend has built in stuff to do
this because it is more than just ETL (it is an ESB where ETL is just a
use case)

On Saturday, January 11, 2014 1:47:40 AM UTC-5, David Pilato wrote:

IMHO, some options:

  • ETL tool like Talend
  • a batch process you need to write
  • modify application which create those data and push to Elasticsearch
    from it

I prefer the latter as your search engine will be well synchronized with
your database.
Also, it will be a gain, database performance wise, as you don't need to
read again your DB to inject into ES.

My 2 cents

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 11 janv. 2014 à 04:12, jsp <jayasu...@gmail.com <javascript:>> a
écrit :

Thanks Karl,
Do you have any recommendation for alternatives to using rivers. i want
to read data from sql db/tables.
thanks
jay

On Friday, 10 January 2014 13:55:29 UTC-8, Karol Gwaj wrote:

Consider the fact that rivers could be deprecated in future versions of
elasticsearch:

Sign Up | LinkedIn

On Wednesday, January 8, 2014 4:26:58 PM UTC, jsp wrote:

Hi,
I am looking at implementing ES to index & query data that I get from my
SQL Server databases/tables. I was initially using river to read data from
Sql server tables but one of the developers in my team recommended looking
at using logstash. Can anyone comment to any benefits of using one over
another? I have not been able to find any documentation regarding reading
data from SQL server using logstash http://logstash.net/docs/1.3.2/.
Can someone point me to a guide on how to get started with logstash & sql
server.
Thanks
J

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/890a3acc-74f9-4999-b42d-790c78a97c39%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cf23220-b281-4b38-92c8-bcbfb8dd3df5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.