How to store only limited number of docs

Elastic Stack Elasticsearch
1

Hello -

I will be collecting data for every 10sec and hence my size of the particular index will be growing. Is there any way to limit the storage. I.e at any point of time i may require the latest 100 docs.count

2

You can use the rollover API in elasticsearch to restrict an index to a particular size. Once that limit is crossed, a new index will be created along with the alias. You can than safely remove the older index.

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-rollover-index.html

I believe this is something which you were looking for.

3

Yes, it matches but i still have few concrens

  • if the size/count of docs reaches 1000/10mb then a new new alias is created with different name and all the docs will be moved to that allias_new_index. But i will doing some operation on existing_index name so at a moment during the when new_index created there will no docs in existing_index any my operation fails? so how to handle this?
    I need to retain the existing_name and at least latest 50~ docs so that any point of time, i will have the existing _index and min of 50 docs to perform my operation.
4

any clue ??

5

There are 2 operations expected for ES, i.e. Read & Write.

For Read operations, in order to retrieve data from old + new index, you can move with either of the solutions.

  1. Use wild card indices while fetching data, so for the indices you can query the index logs_*.
  2. Attach another read alias to multiple index, you need to issue index alias call here.

For Write operation, you should be using the newly attached alias from the rollover operation. Hope this answers to your question.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.