Incremental counter in elasticsearch

(Jayme Rotsaert) #1

I have a +/- large index of 200M documents, the content inside can be seen as a representation of a "user". One user/ JSON document can get fairly large.

I have a daily cron that does a search for each of the users in the index,
it searches for other "users" that fall in his/ her filter and orders them by activeness/ attractiveness, nearby filters.

I want to keep track on how much a certain user has already been "used" / "seen" in other peoples searches.. so most logical would be to keep track of a field "usage" field that increments each time a user is found/ returned.
This data can be volatile as the counter can be reset every day.

There are a few drawbacks / disadvantages though :

  • a complete user document needs to be rewritten each time that field gets incremented /
  • searches will go slower as the complete index will be rewritten each day / complete iteration.
  • need to manually flush the counter value each day when starting a new iteration..

Would want to use parent/ child for this, but apparently you can't sort on "child_values" in elasticsearch + in a sense this also written to the same index corpus starting from ES 6.0.

What's the best approach in solving something like this?
Write a custom plugin and keep values like this in memory?


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.