# What tools to use for indexing ES in production? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 16, 2015, 9:28am UTC](https://discuss.elastic.co/t/what-tools-to-use-for-indexing-es-in-production/22183 "2015-02-16T09:28:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Kevin\_Liu\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_liu_2/32/894_2.png) [@Kevin\_Liu\_2](https://discuss.elastic.co/u/Kevin_Liu_2)\ **Post date:** [February 16, 2015, 9:28am UTC](https://discuss.elastic.co/t/what-tools-to-use-for-indexing-es-in-production/22183/1 "2015-02-16T09:28:33Z")
We want to index our sales records as they come in from our apps. We are using quartz job right now, which is really slow and not really real-time. We will be implementing a message bus soon for firing sales event. The process is: read from a message queue grab some extra data from mysql do some ETL to construct the document index it to ES. I've been reading about apache spark, ES river, logstash. My questions is what kind of tools are right for the job here? is apache spark an over kill? is DIY a better option here? what are you guys using? please advice and point me to the right thing to read. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 16, 2015, 10:30am UTC](https://discuss.elastic.co/t/what-tools-to-use-for-indexing-es-in-production/22183/2 "2015-02-16T10:30:13Z")
This depends on how the app is made and what options you have to extract data from it. On 16 February 2015 at 20:28, Kevin Liu [kevin@ticketfly.com](mailto:kevin@ticketfly.com) wrote: > We want to index our sales records as they come in from our apps. We are > using quartz job right now, which is really slow and not really real-time. > > We will be implementing a message bus soon for firing sales event. > > The process is: > read from a message queue > grab some extra data from mysql > do some ETL to construct the document > index it to ES. > > I've been reading about apache spark, ES river, logstash. > > My questions is what kind of tools are right for the job here? > is apache spark an over kill? > is DIY a better option here? > what are you guys using? > > please advice and point me to the right thing to read. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_Ggo6O0fTFYfH2ggFiPODZCNU0XmVXNBUNK5XeBBSXsg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_Ggo6O0fTFYfH2ggFiPODZCNU0XmVXNBUNK5XeBBSXsg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Kevin\_Liu\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kevin_liu_2/32/894_2.png) [@Kevin\_Liu\_2](https://discuss.elastic.co/u/Kevin_Liu_2)\ **Post date:** [February 19, 2015, 9:07am UTC](https://discuss.elastic.co/t/what-tools-to-use-for-indexing-es-in-production/22183/3 "2015-02-19T09:07:34Z")
Well, the sales messages are coming through kafka. We need to extract some info from the database. We can do anything really. I'm just not sure what are the common practice here. It seems to be so many options. what kind of questions am I not asking here? On Monday, February 16, 2015 at 2:30:45 AM UTC-8, Mark Walkom wrote: > This depends on how the app is made and what options you have to extract > data from it. > > On 16 February 2015 at 20:28, Kevin Liu \<[ke...@ticketfly.com](mailto:ke...@ticketfly.com) \ > > > wrote: > > > We want to index our sales records as they come in from our apps. We are > > using quartz job right now, which is really slow and not really real-time. > > > > We will be implementing a message bus soon for firing sales event. > > > > The process is: > > read from a message queue > > grab some extra data from mysql > > do some ETL to construct the document > > index it to ES. > > > > I've been reading about apache spark, ES river, logstash. > > > > My questions is what kind of tools are right for the job here? > > is apache spark an over kill? > > is DIY a better option here? > > what are you guys using? > > > > please advice and point me to the right thing to read. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4a11083b-4e9f-4be2-9a8e-83caf34d56b4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a11083b-4e9f-4be2-9a8e-83caf34d56b4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 19, 2015, 10:53pm UTC](https://discuss.elastic.co/t/what-tools-to-use-for-indexing-es-in-production/22183/4 "2015-02-19T22:53:34Z")
## Kafka is fine, you could also use Logstash to read this data and send to ES > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... NB this is in the 1.5 BETA release, so use with caution. Otherwise take a look at other Logstash inputs and see if there is something suitable you can leverage, there is also a number of official clients if you want to roll your own. There isn't an official method, but those last two are pretty common. On 19 February 2015 at 20:07, Kevin Liu [kevin@ticketfly.com](mailto:kevin@ticketfly.com) wrote: > Well, the sales messages are coming through kafka. We need to extract some > info from the database. We can do anything really. I'm just not sure what > are the common practice here. It seems to be so many options. what kind of > questions am I not asking here? > > On Monday, February 16, 2015 at 2:30:45 AM UTC-8, Mark Walkom wrote: > > > This depends on how the app is made and what options you have to extract > > data from it. > > > > On 16 February 2015 at 20:28, Kevin Liu [ke...@ticketfly.com](mailto:ke...@ticketfly.com) wrote: > > > > > We want to index our sales records as they come in from our apps. We are > > > using quartz job right now, which is really slow and not really real-time. > > > > > > We will be implementing a message bus soon for firing sales event. > > > > > > The process is: > > > read from a message queue > > > grab some extra data from mysql > > > do some ETL to construct the document > > > index it to ES. > > > > > > I've been reading about apache spark, ES river, logstash. > > > > > > My questions is what kind of tools are right for the job here? > > > is apache spark an over kill? > > > is DIY a better option here? > > > what are you guys using? > > > > > > please advice and point me to the right thing to read. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/949b0c40-e8b1-4d40-bced-68e5c005c713%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4a11083b-4e9f-4be2-9a8e-83caf34d56b4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a11083b-4e9f-4be2-9a8e-83caf34d56b4%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/4a11083b-4e9f-4be2-9a8e-83caf34d56b4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4a11083b-4e9f-4be2-9a8e-83caf34d56b4%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8r2as%3DXJRfYa1cTV3ZGsV\_ckfjW-ZxqQCF1iDa9mz-mA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8r2as%3DXJRfYa1cTV3ZGsV_ckfjW-ZxqQCF1iDa9mz-mA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:31am UTC](https://discuss.elastic.co/t/what-tools-to-use-for-indexing-es-in-production/22183/5 "2017-07-06T00:31:22Z")