Is there any correlation between ttl and version conflicts?

Hi folks,

I'm using elasticsearch to store and analyse logs. Since I wanted old logs to be deleted automatically I added a ttl to my mapping. Now I sometimes get version conflict exceptions when my (PHP) application tries to update a timestamp in one of the fields. I'm trying to update the field using cURL and because of session locking it is impossible for a single user to generate more than one curl_exec()-call at a time. Furthermore I do not provide version information explicidly in the update query. There are two elasticsearch servers handling the log index however all queries are handled by only one of them. The other one is just a standby. The exceptions were thrown only after setting a ttl so I was wondering if there might be any correlation? Can any of you shed some light on that matter?

Best regards, Sebastian

Hi Sebastien,

I believe this could indeed be related, eg.

  1. The update API gets the current version of the document (V1)
  2. Elasticsearch deletes the document because of its TTL, which increments
    the version of the document to V2 > V1
  3. Elasticsearch builds the updated document and adds it to the index: it
    expects the current version in the index to be V1 to succeed, yet it is now
    V2 because of the TTL and thus the update fails.

For your information, it is generally a better practice to evict old logs
to have time-based indices (eg. one index per week) and to remove whole
indices when they become too old. This would be lighter for elasticsearch
as removing an old index is just about removing files on the filesystem
while deleteing lots of documents can trigger heavy merges in order to
reclaim the disk space.

On Thu, Oct 30, 2014 at 1:38 PM, Sebastian sebastianappel@gmx.de wrote:

Hi folks,

I'm using elasticsearch to store and analyse logs. Since I wanted old logs
to be deleted automatically I added a ttl to my mapping. Now I sometimes
get
version conflict exceptions when my (PHP) application tries to update a
timestamp in one of the fields. I'm trying to update the field using cURL
and because of session locking it is impossible for a single user to
generate more than one curl_exec()-call at a time. Furthermore I do not
provide version information explicidly in the update query. There are two
elasticsearch servers handling the log index however all queries are
handled
by only one of them. The other one is just a standby. The exceptions were
thrown only after setting a ttl so I was wondering if there might be any
correlation? Can any of you shed some light on that matter?

Best regards, Sebastian

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Is-there-any-correlation-between-ttl-and-version-conflicts-tp4065542.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1414672713198-4065542.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7UByw7q1NjhF-E3PB4%3DJzRAx8VMZkh8mhmRvgRYGujqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.