# Historical data vs Last **URL:** \ **Category:** Elasticsearch\ **Created:** [February 11, 2016, 7:25pm UTC](https://discuss.elastic.co/t/historical-data-vs-last/41552 "2016-02-11T19:25:38Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![rob.chartier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rob.chartier/32/7743_2.png) [@rob.chartier](https://discuss.elastic.co/u/rob.chartier)\ **Post date:** [February 11, 2016, 7:25pm UTC](https://discuss.elastic.co/t/historical-data-vs-last/41552/1 "2016-02-11T19:25:38Z")
Hi All, We are trying to model an IoT scenario and find that we may have two needs for most of our sensor indices and I wanted to throw it out there to see what others think and if they have already solved this issue. Lets take a simple scenario, a WiFi temperature sensor. It reports data 10 times every second. For each sensor we have location data, the temperature reading, the name of the sensor and the datetime in UTC of the event. Our indexing plan is... Index 1 (Historical): Use \_id = null, to get the historical index built. Index 2 (Last): Use \_id = "the name of the sensor itself", to keep track of the most recent value for the sensor. I understand that for most situations we can easily query the Historical index for the last by version or DateTime with a size of 1 and we can get the most recent value. There are two "issues" that I see: 1. We want to use Kibana to show the LAST reading on a Map, and not see any historical data. Since, in Kibana, we cant express our custom "Last" filter we will not be able to achieve this. Is there a way to achieve this, easily? 1. We think, querying by \_id on the the "Last" index will outperform querying the "Historical" with a Size of 1. Anyone have any concrete evidence either way? Questions, comments, concerns all welcome. Cheers, -Rob ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 12, 2016, 8:30am UTC](https://discuss.elastic.co/t/historical-data-vs-last/41552/2 "2016-02-12T08:30:54Z")
KB currently can't do that, I think there is a FR for it though. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 12, 2016, 8:57am UTC](https://discuss.elastic.co/t/historical-data-vs-last/41552/3 "2016-02-12T08:57:10Z")
Storing the current state, e.g. most recent entry, in a separate index like you are proposing is a very efficient way to get to it from Kibana and something I have seen used in the past. You are basically trading a bit more work at indexing time for a much more efficient query, which is often a good tradeoff. The only thing that concerns me is the high update rate. If may be worthwhile throttling this somehow, especially considering data by default is only flushed and made searchable once per second by default. ---
**Author:** ![rob.chartier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rob.chartier/32/7743_2.png) [@rob.chartier](https://discuss.elastic.co/u/rob.chartier)\ **Post date:** [February 12, 2016, 9:04pm UTC](https://discuss.elastic.co/t/historical-data-vs-last/41552/4 "2016-02-12T21:04:05Z")
@christian The example given is a bit contrived. We actually will be looking potentially thousands to millions of devices with about 50-60 sensors on each, of which about 10-20 update as frequent as about one message every 10 seconds. @workolm By FR, do you mean PR? As in a pull request...? So our initial plan of manually building a "last" index still stands. Thanks again! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:16pm UTC](https://discuss.elastic.co/t/historical-data-vs-last/41552/5 "2017-07-05T23:16:42Z")