I have a question about "Write Thread Pool."

Elastic Stack Elasticsearch
1

Hello.
I'm not good at English. I'm sorry, but please understand.

We've installed our own custom plug-in in ElasticSearch.

We implemented the AbstractTokenFilterFactory.

the "incrementToken" method calls external APIs, with a delay of 20 ms.....

We, the fast index performance is important, so we looked at various ways.

"IncrementToken" reviewed the possibility of asynchronous processing of logic calling external API, but failed to find the correct answer.

It runs within the "Write Thread Pool," and we know that if our CPU is 16 cores, perhaps the maximum Thread Pool is 16+1.

My question is three.

  1. Is it reasonable to set the "Write Thread Pool" to its maximum(core + 1)?

  2. If our CPU is "Hyper-Threading" supported, can 16core CPUs operate as "Write Thread Pool" in "32+1"?

  3. Is there a better way?

2

I would always try to prevent doing API calls at index time. Another option would be to write your own ingest processor, as you can scale them separately by adding dedicated ingest nodes. See https://www.elastic.co/guide/en/elasticsearch/reference/6.6/ingest.html

That said, the same problem remains, that you might end up with the threadpool being full because of a slow API. You can increase the thread pool size, but that would just delay the problem. Having a queue in your ingestion process or doing this as a preprocessing step (like the ingest node but in your own app) might be viable alternatives as well. I would always try to not block Elasticsearch threads if possible, especially the write thread pool.

Hope this helps!

1 Like
3

Thank you very much.

We'll check "Ingest Node."

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.