# Elastic search index strategies under high traffic **URL:** \ **Category:** Elasticsearch\ **Created:** [September 23, 2013, 4:18pm UTC](https://discuss.elastic.co/t/elastic-search-index-strategies-under-high-traffic/13726 "2013-09-23T16:18:59Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Onur\_BARAN](https://avatars.discourse-cdn.com/v4/letter/o/848f3c/32.png) [@Onur\_BARAN](https://discuss.elastic.co/u/Onur_BARAN)\ **Post date:** [September 23, 2013, 4:18pm UTC](https://discuss.elastic.co/t/elastic-search-index-strategies-under-high-traffic/13726/1 "2013-09-23T16:18:59Z")
We use ElasticSearch for our tool's real time metrics and analytics part. ElasticSearch is very cool and fast when we are query our data. (statiticial facets and terms facet) But we have problem when we try to index our hourly data. We collect every our metric data from other services. First we collect data from other services and save them RabbitMQ process. But when queue worker runs our all hourly data not index to ES. Usually %40 of data index in ES and other them lost. So what is your idea about when index ES under high traffic ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [September 23, 2013, 8:50pm UTC](https://discuss.elastic.co/t/elastic-search-index-strategies-under-high-traffic/13726/2 "2013-09-23T20:50:14Z")
Hi Onur, On Mon, Sep 23, 2013 at 6:18 PM, Onur BARAN [baranonur@gmail.com](mailto:baranonur@gmail.com) wrote: > We use Elasticsearch for our tool's real time metrics and analytics part. > Elasticsearch is very cool and fast when we are query our data. > (statiticial facets and terms facet) > > But we have problem when we try to index our hourly data. We collect every > our metric data from other services. First we collect data from other > services and save them RabbitMQ process. But when queue worker runs our all > hourly data not index to ES. Usually %40 of data index in ES and other them > lost. > > So what is your idea about when index ES under high traffic ? > > Even under high traffic, ES shouldn't lose documents, can you give more > details about the errors you are seeing? -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Onur\_BARAN](https://avatars.discourse-cdn.com/v4/letter/o/848f3c/32.png) [@Onur\_BARAN](https://discuss.elastic.co/u/Onur_BARAN)\ **Post date:** [September 24, 2013, 12:44pm UTC](https://discuss.elastic.co/t/elastic-search-index-strategies-under-high-traffic/13726/3 "2013-09-24T12:44:14Z")
There is no error in logs and index result. I try to index dummy document (100.000 doc) When indexing process finished, ES index only ~70000 docs. 23 Eylül 2013 Pazartesi 19:18:59 UTC+3 tarihinde Onur BARAN yazdı: > We use Elasticsearch for our tool's real time metrics and analytics part. > Elasticsearch is very cool and fast when we are query our data. > (statiticial facets and terms facet) > > But we have problem when we try to index our hourly data. We collect every > our metric data from other services. First we collect data from other > services and save them RabbitMQ process. But when queue worker runs our all > hourly data not index to ES. Usually %40 of data index in ES and other them > lost. > > So what is your idea about when index ES under high traffic ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 24, 2013, 1:19pm UTC](https://discuss.elastic.co/t/elastic-search-index-strategies-under-high-traffic/13726/4 "2013-09-24T13:19:20Z")
You should check if the doc id's overlap. If so, you'll accidentally overwrite previous docs. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:14am UTC](https://discuss.elastic.co/t/elastic-search-index-strategies-under-high-traffic/13726/5 "2017-07-06T02:14:55Z")
Closed on July 6, 2017, 2:14am UTC