ES 2.1 index refresh interval not working as expected

Elastic Stack Elasticsearch
1

Hello,

I have the following index:

[root@jupiter ventus]# curl -XGET 'http://jupiter:9200/190_0?pretty=true'
{
"190_0" : {
"aliases" : { },
"mappings" : {
"190" : {
"dynamic" : "strict",
"_all" : {
"enabled" : false
},
"_timestamp" : {
"enabled" : false
},
"_ttl" : {
"enabled" : true
},
"properties" : {
"cacheId" : {
"type" : "string",
"index" : "no"
},
"client_code" : {
"type" : "string",
"index" : "not_analyzed"
},
"creationTime" : {
"type" : "long"
},
"criterion_code" : {
"type" : "string",
"index" : "not_analyzed"
},
"hotel_code" : {
"type" : "string",
"index" : "not_analyzed"
},
"room" : {
"type" : "nested",
"properties" : {
"room_adults" : {
"type" : "integer"
},
"room_children" : {
"type" : "integer"
}
}
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1449158274798",
"indices" : {
"ttl" : {
"interval" : "30s"
}
},
"uuid" : "74l_yrQ6TyWxg38vC3Qfbg",
"store" : {
"type" : "default"
},
"number_of_replicas" : "0",
"number_of_shards" : "1",
"refresh_interval" : "10000ms",
"version" : {
"created" : "2000099"
}
}
},
"warmers" : { }
}
}

Now I index 3 documents correctly:

[root@jupiter ventus]# curl -XPUT 'http://jupiter:9200/190_0/190/3' -d '{ "cacheId" : "3" }'
[root@jupiter ventus]# curl -XPUT 'http://jupiter:9200/190_0/190/2' -d '{ "cacheId" : "2" }'
[root@jupiter ventus]# curl -XPUT 'http://jupiter:9200/190_0/190/1' -d '{ "cacheId" : "1" }'

This index has a refresh interval of 10 seconds, but:

  1. After indexing the first document (id=3), it becomes visible at search time after 5 seconds.
  2. After indexing the second document (id=2), it's visible at search time after 7 seconds.
  3. After indexing the third document (id=1), it's visible at search time after 3 seconds.

These times change, sometimes after indexing a document it takes just 1 seconds to become visible, sometimes takes 5 seconds, ... but always before 10 seconds.

The search I'm performing every second to check visibility is:
[root@jupiter ventus]# curl -XGET 'http://jupiter:9200/190_0/190/_search?q=*:*&pretty=true'

Why is not ES respecting the 10 seconds refresh interval? I understand that after indexing each document, they not should visible until 10 seconds later.

Something misconfigured in the index?

Thanks.

2

The refresh time is a maximum time.
It could happen that you started to create a segment in memory at X.

Then you inject a doc at X+9s.
Then the refresh happens at X+10s.

So your document will become visible 1s after you have injected your document.

3

TTL is deprecated, you should really move away from it.

4

Yes, I know _ttl is deprecated. What is the alternative? Having an 'expiration' date and create a thread that every 30s searches all expired documents? But isn't this what _ttl is exactly doing?

5

Yes, but:

  1. If you create the cron job yourself then you can pause it. _ttl cannot be paused. Elasticsearch could have grown a way to pause _ttl so that isn't a great excuse.
  2. _ttl causes you to delete your index in a tiny bit at a time which is super inefficient. Its generally more efficient to create an index for all documents that will expire in a period of time and then delete the indexes once all documents have expired. That delete is just a filesystem operation. The deletes from _ttl have to be tombstoned and merged out.

For small indexes _ttl worked fine but for large indexes it causes all kinds of trouble. That makes it a nasty trap because people will rely on it in development and then it'll fall over when they go to production.

6

The alternative is to index the current timestamp in each document, and perform search filter to filter out irrelevant documents.

If time range is large, indices should be organized by range, so old indices can be dropped. Dropping indices is a very fast and does not exercise the cluster.

7

Hello,

Thanks for all responses.

My indexes are already organized by range. For example, if for a certain api method my cache time is 45 minutes, I'm creating 5 indexes that contains 9 minutes of documents. After 45 minutes I'm automatically creating a new index and removing the oldest one.

But my problem is that a user can modify the cache time for a document on-the-fly. That means that most documents will have a cache time of 45 minutes, but I also can have several documents with 7-hour or 20-day cache time. At this moment these documents have been stored in an specific index. And I think my only choice here is using _ttl (or an expiration date, which is the same at the end).

I've been thinking a lot about what to do with these documents that don't have the standard cache time, and I can't get a solution different that one index for them with _ttl enabled.

Any ideas would be really appreciated.