I have one VPS running two PHP applications (resp. production and test/staging environment of one application) that are using Elasticsearch. Until now I only had one cluster with one node, that was shared between both apps (at port 9200). I now need to separate the ES for each app, so I could use different data, indexes, mappings etc. for each of them. And I would still like to run everything on single VPS.
With Puppet I was able to set up two nodes listening at port 9200 and 9201 (two services), but they still seem to be dependent on each other – if I update mapping on one, the other app crashes (without logging anything, that's why it's so hard to debugg). I also tried using different cluster.name for each of them, but the the second cluster has UUID: _na_ and updating mapping and data doesn't work.
I'm new to ES so I'll appreciate any noob help, best practices or pointing in correct direction.
as I wrote, I have 2 copies of the same app running on the server, just each of them with different data (production vs. testing/demo). It's easy to separate serving of the apps (Apache) and have them live in different directories, but I'm not sure how to split the data storage in ES since both apps use the same structure (indices & mappings).
Maybe duplicating and prefixing the indices would be better? On the other hand I feel like separating production from test is more cleaner for times when I want to make changes on one of the version, shut one down for maintenance etc. Plus I'm not sure if it would be possible to "sync" ES with data from my database and not touch the indicies of the other app. Does it make sense, am I clear?
I know the best would be to have different machine for testing server, but that's just not an option right now.
I agree it is good to have a separate environment for test and production, but hosting 2 single node clusters on the same VM is going to make things complex, especially if you're not using any facility to isolate the resources (cpu, ram, disk...).
If you want to use a single cluster for both environments, you can use separate indices (different prefixes for example) and/or use index aliases.
It's definitely not ideal to have multiple nodes on a machine but nor is it forbidden and I can imagine it would make separation between the two environments simpler than trying to isolate everything properly within a single cluster by prefixing the names of everything or some other hack. You can isolate nodes' resources (CPU, RAM, disk, ...) with OS tools if needed since they're just normal processes, but I can also imagine that this isn't really needed in a lightly-loaded use case as long as you bear in mind that as your load grows you may need to separate your environments more carefully.
They definitely should have different cluster.name properties. Maybe it would be more helpful to focus on why your attempt to form two clusters didn't work. Can you elaborate on that? I imagine the details in the logs will explain the problem.
I was able to debug a bit more and found out the second cluster throws master_not_discovered_exception. I tried setting the network.publish_host, but then the system service doesn't even boot (node validation exception). Allowing firewall for 9300:9400 doesn't help.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.