Different thread priorities for indexing vs. searching?

Hi,

ES has different thread pools for indexing, searching, etc. Is it possible to tell ES to set different priority on indexing vs. search requests? I didn't see a way to do that.

Wouldn't that be very valuable, especially for the popular logging use case?

Thanks,
Otis

Monitoring - Log Management - Alerting - Anomaly Detection
Elasticsearch Consulting - Support - Training - http://sematext.com/

A good idea in general, but not a good idea on the JVM.

  • you have to be root to raise priority of threads (on Linux at least), so the question is if lowering thread priorities is of any benefit
  • lowering thread priorities make threads always run slower (see UNIX 'nice')
  • you need to instruct ES users how to map Java threads to OS native threads on the specific hardware, which is flaky when using -XX:ThreadPriorityPolicy e.g. by not-so-well documented JVM flags like
-XX:ThreadPriorityPolicy=42
-XX:JavaPriority10_To_OSPriority=0
-XX:JavaPriority9_To_OSPriority=1
-XX:JavaPriority8_To_OSPriority=2
-XX:JavaPriority7_To_OSPriority=3
-XX:JavaPriority6_To_OSPriority=4
-XX:JavaPriority5_To_OSPriority=5
-XX:JavaPriority4_To_OSPriority=6
-XX:JavaPriority3_To_OSPriority=7
-XX:JavaPriority2_To_OSPriority=8
-XX:JavaPriority1_To_OSPriority=9
  • thread behavior varies from JDK to JDK version, e.g. on Solaris, see https://blogs.oracle.com/dave/entry/java_thread_priority_revisted_in
  • thread management depends also on the JVM implementation
  • I'm not familiar with Windows but I bet it's a totally different story
  • from OS to OS, threads would behave differently, which opens a can of worms

Hi,

Hmmm, I think we are out of sync. I'm not referring to process priorities and "nice". I'm referring to Thread class in Java:
http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html
"Every thread has a priority. Threads with higher priority are executed in preference to threads with lower priority...."

And here are the 3 priorities:
http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html#field_summary

So this is a Java-level thing (and maybe it translates to OS-level processes or threads, which must be different on different OSes, but that's "not my concern" here) and I'm thinking it would be useful to be able to configure:
index pool priority: min
search pool priority: max
etc.

Wouldn't that be helpful and super easy to implement?

Otis

Monitoring - Log Management - Alerting - Anomaly Detection
Elasticsearch Consulting - Support - Training - http://sematext.com/