Embedding elasticsearch into app

Elastic Stack Elasticsearch
1

Hello,

We are distributing ES 2.1 within our product. So ES is embbeded into our application. We are seeing several posts where it's told that embedding ES is not a good idea for production environments.

Is this really not recommended? Why? What are the issues with embedding ES?

Thanks.

2

Depends how you embed it really, can you provide more info around that?

3

ES, at least 2.0+, is paranoid about a lot of things. If you try to run Java in process with it then its quite likely you'll run into the security manager. Or seccomp. Or something like that. Plugins are fine. You have to put permissions files in them now, but its generally ok.

Now if you mean "I wan to bundle ES with my application the same way you'd bundle PostgreSQL" then more power to you.

4

Hi,

My application is a cache of xml documents. I'm using ES to index this cache (currently just a concurrentmap), so the user can send requests to get results or delete documents that are no longer valid.

This app is running on Linux and Tomcat 7. It's currently just one node (server), and I'm adding all ES jars needed in my WEB-INF folder, and I'm starting the ES node as follows:

this.node = NodeBuilder.nodeBuilder().settings(s).local(true).data(true).client(false).node();
this.client = this.node.client();

When we develop the cluster version, every app node will be an ES node.

All tests I'm performing are giving good results, and the only issue I'm finding is with marvel (I can't see the cluster from marvel, I need to explore this a bit more).

Why embedded?

  1. Our app is usually installed on a client server, this client installs hardware and software, so embedding ES I avoid my client to install one more piece of software.

  2. I don't have any GC issues because our app is running under Zing JVM.

  3. Un app update implies stopping it. So I can also update ES when updating my app. No reason to update just ES. If I find a severe bug on ES, I can stop and update my app without problems.

And looking for information about this issue with marvel I came across some posts telling that embedding ES in production is not recommended.

So before taking any step forward, I'd really appreciate any opinion from ES experts.

Thanks.

5

Not sure if the provided info is enough. Any opinion would be very appreciated.

6

This is fine as long as you don't want to add plugins from the classpath. NodeBuilder does no longer support this.

Also keep in mind that ES HTTP port 9200 is enabled by default, so because you are on Tomcat, you probably want to disable ES HTTP.

What I don't understand is "when we develop the cluster version" because you disabled cluster mode with local(true).

You will not see optimal performance since ES/Tomcat will share heap and resources, also JVM start parameters. This can soon lead to congestion and scaling issues.

Also note, there is an aging transport plugin/application called elasticsearch-wares which is basically a servlet dispatching requests to and from ES REST API. Once this was the preferred method to embed ES into Java EE. This seems no longer be supported in the near future, yes. See more at Remove wares transport plugin · Issue #12721 · elastic/elasticsearch · GitHub

7

What I don't understand is "when we develop the cluster version" because you disabled cluster mode with local(true).

What I meant was that the cluster version is not developed yet, so currently our app is running on a single node. Ok, it seems there is some quorum that running ES inside the same Tomcat container together with an app can be tricky. So I will install ES as a service, it will be a bit complicated to manage with some clients (one of the main reasons for embedding it) but if this is the best choice ... Thanks for all comments.