# \[hadoop\] newbie question **URL:** \ **Category:** Elasticsearch\ **Created:** [May 3, 2015, 9:21am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520 "2015-05-03T09:21:35Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [May 3, 2015, 9:21am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/1 "2015-05-03T09:21:35Z")
hi, i have few basic questions about es-hadoop, and i would really appreciate your kind help 1. if i have currently ES cluster, do i have motivation to add hadoop layer? 2. is the idea of ES-hadoop, that hadoop will be the data store, and ES the search engine above it? 3. can logstash write to hadoop? 4. when i run queries to ES, does it go to HDFS in real time? thanks a lot! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 3, 2015, 10:01am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/2 "2015-05-03T10:01:38Z")
1. Maybe, depends on your use case. 2. No, they connect but ES does not store data on HDFS 3. Not natively. eg [http://www.devopsa.net/2014/04/three-way-to-use-logstash-with-hadoop.html](http://www.devopsa.net/2014/04/three-way-to-use-logstash-with-hadoop.html) 4. Can you elaborate here, what do you mean (though see 2)? On 3 May 2015 at 19:21, Lior Goldemberg [liorg2@gmail.com](mailto:liorg2@gmail.com) wrote: > hi, > > i have few basic questions about es-hadoop, > and i would really appreciate your kind help > > 1. if i have currently ES cluster, do i have motivation to add hadoop > layer? > > 2. is the idea of ES-hadoop, that hadoop will be the data store, and ES > the search engine above it? > > 3. can logstash write to hadoop? > > 4. when i run queries to ES, does it go to HDFS in real time? > > thanks a lot! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8gTi7kpTxEgN\_\_Z7zrr2vDXOY%3DsdNmZ6XHC3xazkYxTA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8gTi7kpTxEgN__Z7zrr2vDXOY%3DsdNmZ6XHC3xazkYxTA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [May 3, 2015, 10:26am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/3 "2015-05-03T10:26:05Z")
To add to Mark's answer: 1. Hadoop means a lot of things so typically, if you are not familiar with it or not a user, the answer tends to be no 2. No. Data is indexed from Hadoop to Elasticsearch or vice-versa. see [Elasticsearch for Hadoop | Elastic](http://elastic.co/hadoop) and the various presentations on this topic. Again, es-hadoop is meant for Hadoop users trying to leverage Elasticsearch 3. Mark already replied 4. No - see 2. Note that HDFS is an archiving store so accessing in "real-time" means slow access especially for random access. On Sun, May 3, 2015 at 1:01 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > Maybe, depends on your use case. > No, they connect but ES does not store data on HDFS > Not natively. eg > [http://www.devopsa.net/2014/04/three-way-to-use-logstash-with-hadoop.html](http://www.devopsa.net/2014/04/three-way-to-use-logstash-with-hadoop.html) > Can you elaborate here, what do you mean (though see 2)? > > On 3 May 2015 at 19:21, Lior Goldemberg [liorg2@gmail.com](mailto:liorg2@gmail.com) wrote: > > > hi, > > > > i have few basic questions about es-hadoop, > > and i would really appreciate your kind help > > > > 1. if i have currently ES cluster, do i have motivation to add hadoop > > layer? > > > > 2. is the idea of ES-hadoop, that hadoop will be the data store, and ES > > the search engine above it? > > > > 3. can logstash write to hadoop? > > > > 4. when i run queries to ES, does it go to HDFS in real time? > > > > thanks a lot! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cf3e6331-2ac2-4967-89b9-a60f2967b8de%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8gTi7kpTxEgN\_\_Z7zrr2vDXOY%3DsdNmZ6XHC3xazkYxTA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8gTi7kpTxEgN__Z7zrr2vDXOY%3DsdNmZ6XHC3xazkYxTA%40mail.gmail.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmcDZ%2BZGjCkJX46iBYX4tEurRSqR1kScfxV-tRLxW3vD0w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmcDZ%2BZGjCkJX46iBYX4tEurRSqR1kScfxV-tRLxW3vD0w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [May 3, 2015, 10:46am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/4 "2015-05-03T10:46:13Z")
> thanks guys, > > i have an app that needs to write big data, currently writes directly to > ES. > > also i have really heavy aggregations (scripted metric), which takes a > long time (few min) > since i know that ES supposed to be a search engine and not DB (by their > claim), i started to look for Solutions, and i thought of the following > form: 1. write to HDFS 2. index the data after manipulations, that will save me expensive aggregations 3.run queries on ES does it make sense? you know a better scalable option? Lior -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [May 3, 2015, 11:03am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/5 "2015-05-03T11:03:34Z")
Yes that works. It looks like you are only using HDFS and none of the computational components of Hadoop (Map/Reduce, Hive, Spark, etc...) thus you could just import the data from HDFS to Elasticsearch with or without Hadoop. With Hadoop you get parallelism but you need to learn (if you haven't already) one of the compute frameworks out there (there are plenty of options). Without it you can get a potentially easier solution that doesn't parallelize the input (which tends to be a problem only after a certain size). What option fits your scenario depends on your requirements really. I don't want to stray you away from Hadoop (I'm the lead of es-hadoop project) rather point out that it is not the only solution out there and that it comes with a cost. Cheers, On Sun, May 3, 2015 at 1:46 PM, Lior Goldemberg [liorg2@gmail.com](mailto:liorg2@gmail.com) wrote: > > thanks guys, > > > > i have an app that needs to write big data, currently writes directly to > > ES. > > > > also i have really heavy aggregations (scripted metric), which takes a > > long time (few min) > > > since i know that ES supposed to be a search engine and not DB (by their > > claim), i started to look for Solutions, and i thought of the following > > form: > > 1. write to HDFS > 2. index the data after manipulations, that will save me expensive > aggregations > 3.run queries on ES > > does it make sense? > you know a better scalable option? > > Lior > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmfXyMrvTkqiB5A0CZKWq2DgNWxScen7bT97\_y8fskACPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmfXyMrvTkqiB5A0CZKWq2DgNWxScen7bT97_y8fskACPQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 3, 2015, 11:08am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/6 "2015-05-03T11:08:36Z")
ES can do aggregations very quickly, it really depends on what you want to do. I'd suggest trying ES and seeing if it can do what you want, and go from there. On 3 May 2015 at 20:46, Lior Goldemberg [liorg2@gmail.com](mailto:liorg2@gmail.com) wrote: > thanks guys, > > > i have an app that needs to write big data, currently writes directly to > > ES. > > > > also i have really heavy aggregations (scripted metric), which takes a > > long time (few min) > > > since i know that ES supposed to be a search engine and not DB (by their > > claim), i started to look for Solutions, and i thought of the following > > form: > > 1. write to HDFS > 2. index the data after manipulations, that will save me expensive > aggregations > 3.run queries on ES > > does it make sense? > you know a better scalable option? > > Lior > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7aafd5c5-cae4-409d-90d7-719ec78155d3%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_Sv8DV6vX0SZM86H-D4UQAM%3DBXipq9vuJrjZc1qYNaJQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_Sv8DV6vX0SZM86H-D4UQAM%3DBXipq9vuJrjZc1qYNaJQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [May 3, 2015, 11:32am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/7 "2015-05-03T11:32:47Z")
Hi guys, Thanks again for the quick replies, very much appreciated!! We are using ES for the past year and from day 1 we haven’t had good perforce for groovy script that use scripted metric aggregations. Our data is not huge yet, we have 163 indices, 356 shards and 360M documents but when we run the groovy script it can take up to 2-3 minutes. From our understanding it should run much faster. thus we are afraid from the future, when data becomes lots bigger, after the beta stage now i'm not sure whats the different between Hadoop and HDFS. is Hadoop an engine that runs over HDFS? btw, my complicated scenario, is that i have tons of events, with fields: event type, user id, date,.. [lots more]... for example : { userid:1, event\_type:A,date:03/05/2015 14:25:01} { userid:1, event\_type:T,date:03/05/2015 14:25:02} { userid:1, event\_type:S,date:03/05/2015 14:25:03} { userid:1, event\_type:Z,date:03/05/2015 14:25:04} { userid:1, event\_type:B,date:03/05/2015 14:25:05} in the query, i need to find specific flows of users, and not necessary in a roll , for example: A-\>S-\>Z needs to return the user above, wither all the relevant docs. when using scripted metric aggregation, it takes a long time, and moreover- takes lots of memory, and sometimes kill the ES can Hadoop help me with it? i thought of creating a list of events per user (currently i have a type "events" in a daily index, with list of events ordered by date and time, and the user is a field in this type ) thanks again! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b4fba6db-18ca-4365-af8e-9f95a8571075%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b4fba6db-18ca-4365-af8e-9f95a8571075%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [May 3, 2015, 6:37pm UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/8 "2015-05-03T18:37:51Z")
come on guys, you were so helpful so far 🙂 On Sun, May 3, 2015 at 2:32 PM, Lior Goldemberg [liorg2@gmail.com](mailto:liorg2@gmail.com) wrote: > Hi guys, > > Thanks again for the quick replies, very much appreciated!! > > We are using ES for the past year and from day 1 we haven’t had good > perforce for groovy script that use scripted metric aggregations. > > Our data is not huge yet, we have 163 indices, 356 shards and 360M > documents but when we run the groovy script it can take up to 2-3 minutes. > From our understanding it should run much faster. > > thus we are afraid from the future, when data becomes lots bigger, after > the beta stage > > now i'm not sure whats the different between Hadoop and HDFS. > is Hadoop an engine that runs over HDFS? > > btw, my complicated scenario, is that i have tons of events, with fields: > event type, user id, date,.. [lots more]... > > for example : > { userid:1, event\_type:A,date:03/05/2015 14:25:01} > { userid:1, event\_type:T,date:03/05/2015 14:25:02} > { userid:1, event\_type:S,date:03/05/2015 14:25:03} > { userid:1, event\_type:Z,date:03/05/2015 14:25:04} > { userid:1, event\_type:B,date:03/05/2015 14:25:05} > > in the query, i need to find specific flows of users, and not necessary in > a roll , for example: A-\>S-\>Z needs to return the user above, wither all > the relevant docs. > when using scripted metric aggregation, it takes a long time, and > moreover- takes lots of memory, and sometimes kill the ES > > can Hadoop help me with it? > i thought of creating a list of events per user (currently i have a type > "events" in a daily index, with list of events ordered by date and time, > and the user is a field in this type ) > > thanks again! > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/0XCJ1PS2H2o/unsubscribe](https://groups.google.com/d/topic/elasticsearch/0XCJ1PS2H2o/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b4fba6db-18ca-4365-af8e-9f95a8571075%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b4fba6db-18ca-4365-af8e-9f95a8571075%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b4fba6db-18ca-4365-af8e-9f95a8571075%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b4fba6db-18ca-4365-af8e-9f95a8571075%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPYrotrxU5QaPu2A7kBNEhUZxc0J4OnjDTwf3eabRBkhU0%2Bxhw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPYrotrxU5QaPu2A7kBNEhUZxc0J4OnjDTwf3eabRBkhU0%2Bxhw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 3, 2015, 7:15pm UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/9 "2015-05-03T19:15:16Z")
Hi, I am sure Hadoop can help you calculate this, but you may also be able to go about this more efficiently in Elasticsearch. If you, as you mentioned, were to create a user centric index in addition to the event centric one that you have got, you could store a list of all the events belonging to a user there. This would allow you to efficiently identify the users that have all the required events through a simple query, and then just process these to verify that the order is correct, which is likely to scale and perform much better than the current approach. This is what is usually referred to as entity-centric indexing [1]. As updating the user centric index for every event inserted can often be expensive, a common approach is to create a batch job that periodically retrieves all new events, aggregates these per user and updates the user index. This will mean that the user index will not be completely up to date all the time, but as you spread out the processing work, it can make queries much more efficient. [1] [https://www.elastic.co/videos/entity-centric-indexing-london-meetup-sep-2014](https://www.elastic.co/videos/entity-centric-indexing-london-meetup-sep-2014) Best regards, Christian On Sunday, 3 May 2015 10:21:35 UTC+1, Lior Goldemberg wrote: > hi, > > i have few basic questions about es-hadoop, > and i would really appreciate your kind help > > 1. if i have currently ES cluster, do i have motivation to add hadoop > layer? > > 2. is the idea of ES-hadoop, that hadoop will be the data store, and ES > the search engine above it? > > 3. can logstash write to hadoop? > > 4. when i run queries to ES, does it go to HDFS in real time? > > thanks a lot! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/891973ff-14be-4720-9895-d7e6581b2323%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/891973ff-14be-4720-9895-d7e6581b2323%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\ **Post date:** [May 5, 2015, 1:06pm UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/10 "2015-05-05T13:06:51Z")
Thanks a lot Very appreciated On Sunday, May 3, 2015, Christian Dahlqvist [acdahlqvist@gmail.com](mailto:acdahlqvist@gmail.com) wrote: > Hi, > > I am sure Hadoop can help you calculate this, but you may also be able to > go about this more efficiently in Elasticsearch. If you, as you mentioned, > were to create a user centric index in addition to the event centric one > that you have got, you could store a list of all the events belonging to a > user there. This would allow you to efficiently identify the users that > have all the required events through a simple query, and then just process > these to verify that the order is correct, which is likely to scale and > perform much better than the current approach. This is what is usually > referred to as entity-centric indexing [1]. > > As updating the user centric index for every event inserted can often be > expensive, a common approach is to create a batch job that periodically > retrieves all new events, aggregates these per user and updates the user > index. This will mean that the user index will not be completely up to date > all the time, but as you spread out the processing work, it can make > queries much more efficient. > > [1] > [https://www.elastic.co/videos/entity-centric-indexing-london-meetup-sep-2014](https://www.elastic.co/videos/entity-centric-indexing-london-meetup-sep-2014) > > Best regards, > > Christian > > On Sunday, 3 May 2015 10:21:35 UTC+1, Lior Goldemberg wrote: > > > hi, > > > > i have few basic questions about es-hadoop, > > and i would really appreciate your kind help > > > > 1. if i have currently ES cluster, do i have motivation to add hadoop > > layer? > > > > 2. is the idea of ES-hadoop, that hadoop will be the data store, and ES > > the search engine above it? > > > > 3. can logstash write to hadoop? > > > > 4. when i run queries to ES, does it go to HDFS in real time? > > > > thanks a lot! > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/0XCJ1PS2H2o/unsubscribe](https://groups.google.com/d/topic/elasticsearch/0XCJ1PS2H2o/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/891973ff-14be-4720-9895-d7e6581b2323%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/891973ff-14be-4720-9895-d7e6581b2323%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/891973ff-14be-4720-9895-d7e6581b2323%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/891973ff-14be-4720-9895-d7e6581b2323%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPYrotrbMGA6\_sN%3DabqCexW%3DeSw%3DKJ1w2oX2BFbDj-MfVj0biw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPYrotrbMGA6_sN%3DabqCexW%3DeSw%3DKJ1w2oX2BFbDj-MfVj0biw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:15am UTC](https://discuss.elastic.co/t/hadoop-newbie-question/23520/11 "2017-07-06T00:15:37Z")