Scalability of adding custom REST endpoint to ES

Elastic Stack Elasticsearch
1

We are considering extending functionality of elasticsearch by adding a custom plugin and REST endpoint to elasticsearch. The code would read data from the index and build some relationships based on various fields across various documents.

I'm trying to understand weather there will be any issues regarding scalability. Thinks to look out for, gotchas, etc. I couldn't find much on the docs regarding this topic.

2

If you wanted to build this into Elasticsearch then it'd be a plugin. Elasticsearch 5.0 and beyond have some plugin APIs that you can extend, letting you register behavior like extra REST endpoints and extra transport actions, more queries, etc.

One problem is that we break compatibility with these APIs frequently. So a plugin that worked in Elasticsearch 5.0.1 probably wouldn't compile against 5.1.1. For that reason we require that plugins be compile for a specific version of Elasticsearch. It is certainly a pain but we think it is better to blow up at startup in this case then it is to blow up when the plugin is used. At some point this'll be fixed but it'll be a while.

What you are describing would be both a REST action and a Transport action. REST actions are fairly thin wrappers around transport actions, mostly just reading requests and serializing responses. Transport actions have the right dependencies so they can forward requests to the appropriate node. If you are willing to write a transport action you can do pretty much anything so long as you don't run out of memory.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.