How to have multiple nodes on same machine with Elasticsearch 2.0.0

Elastic Stack Logstash
#1

Hi,

I have one machine with ELK (Elasticsearch 2.0.0, Logstash 2.0 and Kibana 4.2.0) stack, but I just have a single node, but I want more 3 nodes.

How can I do this? And how I can define on logstash in waht node will be the created index?

Thanks for help!

#2

Just making sure: Why do want three nodes on the same machine?

And how I can define on logstash in waht node will be the created index?

Logstash itself has no control over this. What problem are you trying to solve?

1 Like
#3

I'm thinking organize indexes by type into different nodes, but maybe no sense.
If I use nodes with different machines, how I can ensure all indexes are in same cluster? How I can use a master node on a machine and slaves with another nodes in other machines?

Thanks for the advices

#4

I'm thinking organize indexes by type into different nodes, but maybe no sense.

No, it's hard to see why that would make sense.

If I use nodes with different machines, how I can ensure all indexes are in same cluster? How I can use a master node on a machine and slaves with another nodes in other machines?

You are free to choose exactly which cluster an Elasticsearch node should be a member of so I don't understand your questions.

#5

My doubt is how to make comunications beetween machines with same cluster, and see all nodes with HQ plugin or Kopf plugin installed on "master" machine (node per machine)

#6

Just use regular unicast discovery and configure each node so that it can find at least one other node that's connected to the rest of the cluster. This procedure is the same regardless of whether you run the nodes on the same machine or not.

discovery.zen.ping.unicast.hosts: hostname1:9300, hostname2:9300
1 Like
#7

Thanks for help magnus, that was my doubt. :wink:

#8

Hi again,

I have this configurations on two machines: machine 1 is the master, and I want see all indexes of all machines in machine 1.
Here are my configuration for elasticsearch in the machines. At the moment I can't see in machine 1 indexes of machine 2.

Machine 1:
---------------------------------- Cluster -----------------------------------
cluster.name: MYCLUSTERNAME
---------------------------------- Network -----------------------------------
network.host: localhost
--------------------------------- Discovery ----------------------------------
discovery.zen.ping.unicast.hosts: ["IP_MACHINE_1:9300"]

Machine 2:
---------------------------------- Cluster -----------------------------------
cluster.name: MYCLUSTERNAME
---------------------------------- Network -----------------------------------
network.host: 127.0.1
--------------------------------- Discovery ----------------------------------
discovery.zen.ping.unicast.hosts: ["IP_MACHINE_1"]

#9

Please post your configuration again without all the irrelevant comments (use e.g. sed 's/^ *#.*//; /^$/d' /etc/elasticsearch/elasticsearch.yml) and format your post as code so that it's readable.

Also, have a look in your Elasticsearch log files. It's likely that it contains clues.

#10

I put just what I uncomment on file elasticsearch.yml in each machine.

#11

With network.host set to 127.0.0.1 or localhost Elasticsearch will only listen on the loopback interface and your two nodes won't be able to contact each other.

#12

So, to each machine maintain communication I have just to comment the line network.host? The configuration in discovery.zen.ping.unicast.hosts is correct?

#13

So, to each machine maintain communication I have just to comment the line network.host?

With ES 2.0 and later I think network.host defaults to localhost so you probably have to set it to e.g. 0.0.0.0 to have it listen on all interfaces. See the release notes. For ES 1.7 and earlier you don't need that setting.

The configuration in discovery.zen.ping.unicast.hosts is correct?

You have the same value for both machines. They should typically point to each other.

1 Like
#15

It works.
Thanks for help Magnus. :slight_smile:

#16

Hi,

Now I have a master node and data node.
But when I create an index, shards are balanced through the nodes.

How I can define in template index, to the all 5 shards stay just in the node where they are created??

Thanks for help.

#17

How I can define in template index, to the all 5 shards stay just in the node where they are created??

Shards can obviously not always stay where they were created so it's not clear what you mean. What problem are you trying to solve?

#18

I have data on master node, but when I join to cluster another node, shards are balanced between the master node and new data node.

What I want is when I create an index in a node, all shards of that index stay in that node.

#19

Yes, I understand your situation but why do you want shards to stay in the original node? What's the point?

#20

Because at this moment I want nodes have just one type of logs.

#21

Okay. What's the point of that? What is the end goal? Having just one type of log per node sounds like a bogus goal.

You could probably use the technique described in https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html to accomplish something similar to what you want.

1 Like