# Performance issues using Elasticsearch as a time window storage **URL:** \ **Category:** Elasticsearch\ **Created:** [September 11, 2013, 1:21pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557 "2013-09-11T13:21:39Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Daniel\_Galinkin](https://avatars.discourse-cdn.com/v4/letter/d/da6949/32.png) [@Daniel\_Galinkin](https://discuss.elastic.co/u/Daniel_Galinkin)\ **Post date:** [September 11, 2013, 1:21pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/1 "2013-09-11T13:21:39Z")
We are using elastic search almost as a cache, storing documents found in a time window. We continuously insert a lot of documents of different sizes and then we search in the ES using text queries combined with a date filter so the current thread does not get documents it has already seen. Something like this: "((word1 AND word 2) OR (word3 AND word4)) AND insertedDate \> 1389000" We maintain the data in the elastic search for 30 minutes, using the TTL feature. Today we have at least 3 machines inserting new documents in bulk requests every minute for each machine and searching using queries like the one above pratically continuously. We are having a lot of trouble indexing and retrieving these documents, we are not getting a good throughput volume of documents being indexed and returned by ES. We can't get even 200 documents indexed per second. We believe the problem lies in the simultaneous queries, inserts and TTL deletes. We don't need to keep old data in elastic, we just need a small time window of documents indexed in elastic at a given time. What should we do to improve our performance? Thanks in advance Machine type: - An Amazon EC2 medium instance (3.7 GB of RAM) Additional information: - The code used to build the index is something like this: [https://gist.github.com/dggc/6523411](https://gist.github.com/dggc/6523411) - Our elasticsearch.json configuration file: [https://gist.github.com/dggc/6523421](https://gist.github.com/dggc/6523421) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [September 11, 2013, 2:14pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/2 "2013-09-11T14:14:25Z")
How strict is this 30 minutes TTL? What would improve you data flow is if you work with multiple indices. You create a new index and then start to index into it. After 30 minutes you create a new index and start into that new index. Then again after 30 minutes you delete the first index and create a new index (third) and strat to index into that. So at most you have 2 indices active and with using aliases you can point to the active indices for read operations. I expect better performance indexing data via this manner. On 11 September 2013 15:21, Daniel Galinkin [danielgalinkin@gmail.com](mailto:danielgalinkin@gmail.com)wrote: > We are using Elasticsearch almost as a cache, storing documents found in > a time window. We continuously insert a lot of documents of different sizes > and then we search in the ES using text queries combined with a date filter > so the current thread does not get documents it has already seen. Something > like this: > > "((word1 AND word 2) OR (word3 AND word4)) AND insertedDate \> 1389000" > > We maintain the data in the Elasticsearch for 30 minutes, using the TTL > feature. Today we have at least 3 machines inserting new documents in bulk > requests every minute for each machine and searching using queries like the > one above pratically continuously. > > We are having a lot of trouble indexing and retrieving these documents, we > are not getting a good throughput volume of documents being indexed and > returned by ES. We can't get even 200 documents indexed per second. > > We believe the problem lies in the simultaneous queries, inserts and TTL > deletes. We don't need to keep old data in elastic, we just need a small > time window of documents indexed in elastic at a given time. > What should we do to improve our performance? > > Thanks in advance > > Machine type: > > - An Amazon EC2 medium instance (3.7 GB of RAM) > > Additional information: > > - The code used to build the index is something like this: > [Code to create the mappings in Java · GitHub](https://gist.github.com/dggc/6523411) > > - Our elasticsearch.json configuration file: > [elastic configuration · GitHub](https://gist.github.com/dggc/6523421) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Galinkin](https://avatars.discourse-cdn.com/v4/letter/d/da6949/32.png) [@Daniel\_Galinkin](https://discuss.elastic.co/u/Daniel_Galinkin)\ **Post date:** [September 11, 2013, 2:38pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/3 "2013-09-11T14:38:52Z")
Thanks for the reply! Talking on the irc channel, the nice guys there also suggested something along these lines. I'm currently implementing this, and will try it soon. On another note: we used to have many mappings inside the same index, and would use these mappings to filter the results for each mapping. We changed that, and now have one index for each type that used to have a mapping. Which is the theoretically better option? On Wednesday, September 11, 2013 11:14:25 AM UTC-3, Martijn v Groningen wrote: > How strict is this 30 minutes TTL? What would improve you data flow is if > you work with multiple indices. You create a new index and then start to > index into it. After 30 minutes you create a new index and start into that > new index. Then again after 30 minutes you delete the first index and > create a new index (third) and strat to index into that. So at most you > have 2 indices active and with using aliases you can point to the active > indices for read operations. I expect better performance indexing data via > this manner. > > On 11 September 2013 15:21, Daniel Galinkin \<[danielg...@gmail.com](mailto:danielg...@gmail.com)\ > > > wrote: > > > We are using Elasticsearch almost as a cache, storing documents found in > > a time window. We continuously insert a lot of documents of different sizes > > and then we search in the ES using text queries combined with a date filter > > so the current thread does not get documents it has already seen. Something > > like this: > > > > "((word1 AND word 2) OR (word3 AND word4)) AND insertedDate \> 1389000" > > > > We maintain the data in the Elasticsearch for 30 minutes, using the TTL > > feature. Today we have at least 3 machines inserting new documents in bulk > > requests every minute for each machine and searching using queries like the > > one above pratically continuously. > > > > We are having a lot of trouble indexing and retrieving these documents, > > we are not getting a good throughput volume of documents being indexed and > > returned by ES. We can't get even 200 documents indexed per second. > > > > We believe the problem lies in the simultaneous queries, inserts and TTL > > deletes. We don't need to keep old data in elastic, we just need a small > > time window of documents indexed in elastic at a given time. > > What should we do to improve our performance? > > > > Thanks in advance > > > > Machine type: > > > > - An Amazon EC2 medium instance (3.7 GB of RAM) > > > > Additional information: > > > > - The code used to build the index is something like this: > > [Code to create the mappings in Java · GitHub](https://gist.github.com/dggc/6523411) > > > > - Our elasticsearch.json configuration file: > > [elastic configuration · GitHub](https://gist.github.com/dggc/6523421) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Met vriendelijke groet, > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [September 11, 2013, 2:54pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/4 "2013-09-11T14:54:22Z")
How many mappings are we talking about? Having a single mapping per index is a bit more optimized, but you can easily have a bunch of types per index. On 11 September 2013 16:38, Daniel Galinkin [danielgalinkin@gmail.com](mailto:danielgalinkin@gmail.com)wrote: > Thanks for the reply! > > Talking on the irc channel, the nice guys there also suggested something > along these lines. I'm currently implementing this, and will try it soon. > > On another note: we used to have many mappings inside the same index, and > would use these mappings to filter the results for each mapping. > We changed that, and now have one index for each type that used to have a > mapping. Which is the theoretically better option? > > On Wednesday, September 11, 2013 11:14:25 AM UTC-3, Martijn v Groningen > wrote: > > > How strict is this 30 minutes TTL? What would improve you data flow is if > > you work with multiple indices. You create a new index and then start to > > index into it. After 30 minutes you create a new index and start into that > > new index. Then again after 30 minutes you delete the first index and > > create a new index (third) and strat to index into that. So at most you > > have 2 indices active and with using aliases you can point to the active > > indices for read operations. I expect better performance indexing data via > > this manner. > > > > On 11 September 2013 15:21, Daniel Galinkin [danielg...@gmail.com](mailto:danielg...@gmail.com) wrote: > > > > > We are using Elasticsearch almost as a cache, storing documents found > > > in a time window. We continuously insert a lot of documents of different > > > sizes and then we search in the ES using text queries combined with a date > > > filter so the current thread does not get documents it has already seen. > > > Something like this: > > > > > > "((word1 AND word 2) OR (word3 AND word4)) AND insertedDate \> 1389000" > > > > > > We maintain the data in the Elasticsearch for 30 minutes, using the TTL > > > feature. Today we have at least 3 machines inserting new documents in bulk > > > requests every minute for each machine and searching using queries like the > > > one above pratically continuously. > > > > > > We are having a lot of trouble indexing and retrieving these documents, > > > we are not getting a good throughput volume of documents being indexed and > > > returned by ES. We can't get even 200 documents indexed per second. > > > > > > We believe the problem lies in the simultaneous queries, inserts and TTL > > > deletes. We don't need to keep old data in elastic, we just need a small > > > time window of documents indexed in elastic at a given time. > > > What should we do to improve our performance? > > > > > > Thanks in advance > > > > > > Machine type: > > > > > > - An Amazon EC2 medium instance (3.7 GB of RAM) > > > > > > Additional information: > > > > > > - The code used to build the index is something like this: > > > [https://gist.github.com/dggc/\*\*6523411](https://gist.github.com/dggc/**6523411)[https://gist.github.com/dggc/6523411](https://gist.github.com/dggc/6523411) > > > > > > - Our elasticsearch.json configuration file: > > > [https://gist.github.com/dggc/\*\*6523421](https://gist.github.com/dggc/**6523421)[https://gist.github.com/dggc/6523421](https://gist.github.com/dggc/6523421) > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- Met vriendelijke groet, Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Galinkin](https://avatars.discourse-cdn.com/v4/letter/d/da6949/32.png) [@Daniel\_Galinkin](https://discuss.elastic.co/u/Daniel_Galinkin)\ **Post date:** [September 11, 2013, 4:04pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/5 "2013-09-11T16:04:49Z")
We currently have 10 mappings. On Wednesday, September 11, 2013 11:54:22 AM UTC-3, Martijn v Groningen wrote: > How many mappings are we talking about? Having a single mapping per index > is a bit more optimized, but you can easily have a bunch of types per > index. > > On 11 September 2013 16:38, Daniel Galinkin \<[danielg...@gmail.com](mailto:danielg...@gmail.com)\ > > > wrote: > > > Thanks for the reply! > > > > Talking on the irc channel, the nice guys there also suggested something > > along these lines. I'm currently implementing this, and will try it soon. > > > > On another note: we used to have many mappings inside the same index, and > > would use these mappings to filter the results for each mapping. > > We changed that, and now have one index for each type that used to have a > > mapping. Which is the theoretically better option? > > > > On Wednesday, September 11, 2013 11:14:25 AM UTC-3, Martijn v Groningen > > wrote: > > > > > How strict is this 30 minutes TTL? What would improve you data flow is > > > if you work with multiple indices. You create a new index and then start to > > > index into it. After 30 minutes you create a new index and start into that > > > new index. Then again after 30 minutes you delete the first index and > > > create a new index (third) and strat to index into that. So at most you > > > have 2 indices active and with using aliases you can point to the active > > > indices for read operations. I expect better performance indexing data via > > > this manner. > > > > > > On 11 September 2013 15:21, Daniel Galinkin [danielg...@gmail.com](mailto:danielg...@gmail.com)wrote: > > > > > > > We are using Elasticsearch almost as a cache, storing documents found > > > > in a time window. We continuously insert a lot of documents of different > > > > sizes and then we search in the ES using text queries combined with a date > > > > filter so the current thread does not get documents it has already seen. > > > > Something like this: > > > > > > > > "((word1 AND word 2) OR (word3 AND word4)) AND insertedDate \> 1389000" > > > > > > > > We maintain the data in the Elasticsearch for 30 minutes, using the > > > > TTL feature. Today we have at least 3 machines inserting new documents in > > > > bulk requests every minute for each machine and searching using queries > > > > like the one above pratically continuously. > > > > > > > > We are having a lot of trouble indexing and retrieving these documents, > > > > we are not getting a good throughput volume of documents being indexed and > > > > returned by ES. We can't get even 200 documents indexed per second. > > > > > > > > We believe the problem lies in the simultaneous queries, inserts and > > > > TTL deletes. We don't need to keep old data in elastic, we just need a > > > > small time window of documents indexed in elastic at a given time. > > > > What should we do to improve our performance? > > > > > > > > Thanks in advance > > > > > > > > Machine type: > > > > > > > > - An Amazon EC2 medium instance (3.7 GB of RAM) > > > > > > > > Additional information: > > > > > > > > - The code used to build the index is something like this: > > > > [https://gist.github.com/dggc/\*\*6523411](https://gist.github.com/dggc/**6523411)[https://gist.github.com/dggc/6523411](https://gist.github.com/dggc/6523411) > > > > > > > > - Our elasticsearch.json configuration file: > > > > [https://gist.github.com/dggc/\*\*6523421](https://gist.github.com/dggc/**6523421)[https://gist.github.com/dggc/6523421](https://gist.github.com/dggc/6523421) > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > Met vriendelijke groet, > > > > > > Martijn van Groningen > > -- > Met vriendelijke groet, > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Daniel\_Galinkin](https://avatars.discourse-cdn.com/v4/letter/d/da6949/32.png) [@Daniel\_Galinkin](https://discuss.elastic.co/u/Daniel_Galinkin)\ **Post date:** [September 17, 2013, 9:43pm UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/6 "2013-09-17T21:43:25Z")
Sorry about the long delay to give you some feedback. Things were kind of hectic here at our company, and I chose to wait for calmer times to give a more detailed account of how we solved our issue. We still have to do some benchmarks to measure the actual improvements, but the point is that we solved the issue 🙂 I posted how we solved our issue in my StackOverflow question, if you are interested, check it out: > On Wednesday, September 11, 2013 1:04:49 PM UTC-3, Daniel Galinkin wrote: > We currently have 10 mappings. > > On Wednesday, September 11, 2013 11:54:22 AM UTC-3, Martijn v Groningen > wrote: > > > How many mappings are we talking about? Having a single mapping per index > > is a bit more optimized, but you can easily have a bunch of types per > > index. > > > > On 11 September 2013 16:38, Daniel Galinkin [danielg...@gmail.com](mailto:danielg...@gmail.com) wrote: > > > > > Thanks for the reply! > > > > > > Talking on the irc channel, the nice guys there also suggested something > > > along these lines. I'm currently implementing this, and will try it soon. > > > > > > On another note: we used to have many mappings inside the same index, > > > and would use these mappings to filter the results for each mapping. > > > We changed that, and now have one index for each type that used to have > > > a mapping. Which is the theoretically better option? > > > > > > On Wednesday, September 11, 2013 11:14:25 AM UTC-3, Martijn v Groningen > > > wrote: > > > > > > > How strict is this 30 minutes TTL? What would improve you data flow > > > > is if you work with multiple indices. You create a new index and then start > > > > to index into it. After 30 minutes you create a new index and start into > > > > that new index. Then again after 30 minutes you delete the first index and > > > > create a new index (third) and strat to index into that. So at most you > > > > have 2 indices active and with using aliases you can point to the active > > > > indices for read operations. I expect better performance indexing data via > > > > this manner. > > > > > > > > On 11 September 2013 15:21, Daniel Galinkin [danielg...@gmail.com](mailto:danielg...@gmail.com)wrote: > > > > > > > > > We are using Elasticsearch almost as a cache, storing documents found > > > > > in a time window. We continuously insert a lot of documents of different > > > > > sizes and then we search in the ES using text queries combined with a date > > > > > filter so the current thread does not get documents it has already seen. > > > > > Something like this: > > > > > > > > > > "((word1 AND word 2) OR (word3 AND word4)) AND insertedDate \> 1389000" > > > > > > > > > > We maintain the data in the Elasticsearch for 30 minutes, using the > > > > > TTL feature. Today we have at least 3 machines inserting new documents in > > > > > bulk requests every minute for each machine and searching using queries > > > > > like the one above pratically continuously. > > > > > > > > > > We are having a lot of trouble indexing and retrieving these > > > > > documents, we are not getting a good throughput volume of documents being > > > > > indexed and returned by ES. We can't get even 200 documents indexed per > > > > > second. > > > > > > > > > > We believe the problem lies in the simultaneous queries, inserts and > > > > > TTL deletes. We don't need to keep old data in elastic, we just need a > > > > > small time window of documents indexed in elastic at a given time. > > > > > What should we do to improve our performance? > > > > > > > > > > Thanks in advance > > > > > > > > > > Machine type: > > > > > > > > > > - An Amazon EC2 medium instance (3.7 GB of RAM) > > > > > > > > > > Additional information: > > > > > > > > > > - The code used to build the index is something like this: > > > > > [https://gist.github.com/dggc/\*\*6523411](https://gist.github.com/dggc/**6523411)[https://gist.github.com/dggc/6523411](https://gist.github.com/dggc/6523411) > > > > > > > > > > - Our elasticsearch.json configuration file: > > > > > [https://gist.github.com/dggc/\*\*6523421](https://gist.github.com/dggc/**6523421)[https://gist.github.com/dggc/6523421](https://gist.github.com/dggc/6523421) > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > > > -- > > > > Met vriendelijke groet, > > > > > > > > Martijn van Groningen > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:15am UTC](https://discuss.elastic.co/t/performance-issues-using-elasticsearch-as-a-time-window-storage/13557/7 "2017-07-06T02:15:53Z")