If I have ELK stack running on EC2. How can I make the ES as a cluster?


(vjbangis) #1

If I have ELK stack running on EC2. How can I make the ES as a cluster?

or I have to create another ELK stack again to make it as a cluster?

TIA!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c90b8417-8568-423e-a592-a151e4b0fa12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(arshpreet singh) #2

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis jessviray0708@gmail.com wrote:

If I have ELK stack running on EC2. How can I make the ES as a cluster?

They say it is super easy to do/find.

--

Thanks
Arshpreet singh

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAstK2GrS%3DnsPXXkOg%3DHMWqU4MibdGaULqD0psx3RQtr1qw_SA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(vjbangis) #3

Thanks for the prompt reply!

Well, I would try that. But I have to make sure that the ELK should always
be the master right?

On Sunday, July 27, 2014 5:30:44 PM UTC+8, arshpreet singh wrote:

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis <jessvi...@gmail.com
<javascript:>> wrote:

If I have ELK stack running on EC2. How can I make the ES as a cluster?

They say it is super easy to do/find.
http://stackoverflow.com/questions/16821101/how-to-set-up-es-cluster

--

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6c1e03e8-ab0c-444f-bc7f-44b5f20c9979%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

Can you elaborate a bit more, do you only have one node now?

It is simple to expand a single node to multiple.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 July 2014 19:30, arshpreet singh arsh840@gmail.com wrote:

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis jessviray0708@gmail.com wrote:

If I have ELK stack running on EC2. How can I make the ES as a cluster?

They say it is super easy to do/find.
http://stackoverflow.com/questions/16821101/how-to-set-up-es-cluster

--

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAAstK2GrS%3DnsPXXkOg%3DHMWqU4MibdGaULqD0psx3RQtr1qw_SA%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bjx_Aeufs84Lsiqo59VmiUEezrjGc9L14xGXqhgd%2BcLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(arshpreet singh) #5

On Sun, Jul 27, 2014 at 3:17 PM, Mark Walkom markw@campaignmonitor.com wrote:

Can you elaborate a bit more, do you only have one node now?

It is simple to expand a single node to multiple.

Who is asking whom? Please use Plain text format for interactions on
mailing list.

--

Thanks
Arshpreet singh

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAstK2Eh%2BvQwkzOO%3DG0j_53dYRqH_pupkMRsg03dAkAYRtN_Pg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(vjbangis) #6

Yup, I have one node acting as ELK stack running on EC2 and I want to add
more nodes.

With that single node (ELK), if I run another EC2 instance and installed ES
only would that make a cluster?

On Sunday, July 27, 2014 5:47:53 PM UTC+8, Mark Walkom wrote:

Can you elaborate a bit more, do you only have one node now?

It is simple to expand a single node to multiple.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 27 July 2014 19:30, arshpreet singh <ars...@gmail.com <javascript:>>
wrote:

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis <jessvi...@gmail.com
<javascript:>> wrote:

If I have ELK stack running on EC2. How can I make the ES as a cluster?

They say it is super easy to do/find.
http://stackoverflow.com/questions/16821101/how-to-set-up-es-cluster

--

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAAstK2GrS%3DnsPXXkOg%3DHMWqU4MibdGaULqD0psx3RQtr1qw_SA%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #7

There's a bit more to it, but yes that is the general idea.

I'm not familiar with ES on EC2 though so I can't give you any directions,
though something like
http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ may help.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 July 2014 20:32, vjbangis jessviray0708@gmail.com wrote:

Yup, I have one node acting as ELK stack running on EC2 and I want to add
more nodes.

With that single node (ELK), if I run another EC2 instance and installed
ES only would that make a cluster?

On Sunday, July 27, 2014 5:47:53 PM UTC+8, Mark Walkom wrote:

Can you elaborate a bit more, do you only have one node now?

It is simple to expand a single node to multiple.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 27 July 2014 19:30, arshpreet singh ars...@gmail.com wrote:

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis jessvi...@gmail.com wrote:

If I have ELK stack running on EC2. How can I make the ES as a cluster?

They say it is super easy to do/find.
http://stackoverflow.com/questions/16821101/how-to-set-up-es-cluster

--

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAAstK2GrS%3DnsPXXkOg%3DHMWqU4MibdGaULqD0psx3RQtr1qw
_SA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b%3D773b-hkkD2gL9s%2BRbtTNfZYGqy5Q1i_cpJaM%3D6PyUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(vjbangis) #8

Actually I'm following that tutorials but according to this link
https://groups.google.com/forum/#!topic/elasticsearch/x8BjtiMTSt4.
The "S3 Gateway" has been dropped, so you'll either need to use EBS, or set
up some mechanism to do snapshots to S3. Other than that, no major changes.

On Sunday, July 27, 2014 6:43:08 PM UTC+8, Mark Walkom wrote:

There's a bit more to it, but yes that is the general idea.

I'm not familiar with ES on EC2 though so I can't give you any directions,
though something like
http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ may help.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 27 July 2014 20:32, vjbangis <jessvi...@gmail.com <javascript:>> wrote:

Yup, I have one node acting as ELK stack running on EC2 and I want to add
more nodes.

With that single node (ELK), if I run another EC2 instance and installed
ES only would that make a cluster?

On Sunday, July 27, 2014 5:47:53 PM UTC+8, Mark Walkom wrote:

Can you elaborate a bit more, do you only have one node now?

It is simple to expand a single node to multiple.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 27 July 2014 19:30, arshpreet singh ars...@gmail.com wrote:

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis jessvi...@gmail.com wrote:

If I have ELK stack running on EC2. How can I make the ES as a
cluster?

They say it is super easy to do/find.
http://stackoverflow.com/questions/16821101/how-to-set-up-es-cluster

--

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAAstK2GrS%3DnsPXXkOg%
3DHMWqU4MibdGaULqD0psx3RQtr1qw_SA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e86c544c-3ff8-45eb-b45c-95c71e1db07c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Aaron Mefford) #9

There are some new options in the latest builds of ElasticSearch as I
understand it that replace the old S3 Gateway.

However, neither the S3 Gateway nor those others are requirements for
setting up ElasticSearch on EC2. They are only disaster recovery options
that will help you to get back up and running in the event of a failure.
Similar capability can be had using EBS and periodic snapshots.

Running ElasticSearch on EC2 feels natural and works well. The Elastic
capabilities of both systems come together very nicely. It is easy to add
and remove nodes to grow and shrink your capacity.

Probably the biggest pitfall is making sure that you give your cluster a
unique name, or you may end up with your nodes joining someone else's
ElasticSearch cluster. I saw this when I brought my first ES node online
on EC2.

The traditional Zen auto discovery will work on EC2, especially if using
VPC where you can control your own subnets. There is also a plugin that
allows using the Amazon instance tags to facilitate auto discovery.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-discovery-ec2.html#modules-discovery-ec2.

Don't get so hung up that the S3 Gateway has been dropped. You can easily
deploy using EBS volumes. If you have large indexes that will not fit in
memory or high indexing rates, you will need to use an EBS Optimized
instance and provisioned IOPS. You can also use multiple EBS volumes and
software raid(mdadm) in order to increase IO, with or without provisioned
IOPS.

On Sunday, July 27, 2014 4:47:20 AM UTC-6, vjbangis wrote:

Actually I'm following that tutorials but according to this link
https://groups.google.com/forum/#!topic/elasticsearch/x8BjtiMTSt4.
The "S3 Gateway" has been dropped, so you'll either need to use EBS, or
set up some mechanism to do snapshots to S3. Other than that, no major
changes.

On Sunday, July 27, 2014 6:43:08 PM UTC+8, Mark Walkom wrote:

There's a bit more to it, but yes that is the general idea.

I'm not familiar with ES on EC2 though so I can't give you any
directions, though something like
http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ may help.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 27 July 2014 20:32, vjbangis jessvi...@gmail.com wrote:

Yup, I have one node acting as ELK stack running on EC2 and I want to
add more nodes.

With that single node (ELK), if I run another EC2 instance and installed
ES only would that make a cluster?

On Sunday, July 27, 2014 5:47:53 PM UTC+8, Mark Walkom wrote:

Can you elaborate a bit more, do you only have one node now?

It is simple to expand a single node to multiple.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 27 July 2014 19:30, arshpreet singh ars...@gmail.com wrote:

On Sun, Jul 27, 2014 at 2:58 PM, vjbangis jessvi...@gmail.com wrote:

If I have ELK stack running on EC2. How can I make the ES as a
cluster?

They say it is super easy to do/find.
http://stackoverflow.com/questions/16821101/how-to-set-up-es-cluster

--

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/

GOD,Nature,Mathematics,Programming.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAAstK2GrS%3DnsPXXkOg%
3DHMWqU4MibdGaULqD0psx3RQtr1qw_SA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b5e04682-f1a1-4799-a977-fd19984c3530%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/964816a3-0b65-4ddc-b0b5-92e933dd0b24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(vjbangis) #10

Thanks to that Aaron.

BTW, should the "data" path be the same size and path with each other
(node/s).

let say
Node* 1*
path.data: /data/elasticsearch
size = 100GB (EBS)

Node 2
path.data: /data/elasticsearch
size = 100GB (EBS)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7b5aa1bc-dc8b-48bd-a2c8-f6ee3ef3e809%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Aaron Mefford) #11

I don't know that ES has any intelligence to support varied node sizes so I
would say yes they should be the same size. I've not looked into this so I
may be wrong.

Also I use multiple Ebs volumes in a software raid. to increase non
provisioned iops. Not necessary if you use piops.

Aaron

Sent from my iPhone

On Aug 2, 2014, at 9:41 AM, vjbangis jessviray0708@gmail.com wrote:

Thanks to that Aaron.

BTW, should the "data" path be the same size and path with each other
(node/s).

let say
Node* 1*
path.data: /data/elasticsearch
size = 100GB (EBS)

Node 2
path.data: /data/elasticsearch
size = 100GB (EBS)

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVs4RKYRGXs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7b5aa1bc-dc8b-48bd-a2c8-f6ee3ef3e809%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7b5aa1bc-dc8b-48bd-a2c8-f6ee3ef3e809%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/-611812713062967954%40unknownmsgid.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #12

ES can take disk space into account, 1.3.X does this automatically -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html#disk

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 4 August 2014 03:36, Aaron Mefford aaron@mefford.org wrote:

I don't know that ES has any intelligence to support varied node sizes so
I would say yes they should be the same size. I've not looked into this so
I may be wrong.

Also I use multiple Ebs volumes in a software raid. to increase non
provisioned iops. Not necessary if you use piops.

Aaron

Sent from my iPhone

On Aug 2, 2014, at 9:41 AM, vjbangis jessviray0708@gmail.com wrote:

Thanks to that Aaron.

BTW, should the "data" path be the same size and path with each other
(node/s).

let say
Node* 1*
path.data: /data/elasticsearch
size = 100GB (EBS)

Node 2
path.data: /data/elasticsearch
size = 100GB (EBS)

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVs4RKYRGXs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7b5aa1bc-dc8b-48bd-a2c8-f6ee3ef3e809%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7b5aa1bc-dc8b-48bd-a2c8-f6ee3ef3e809%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/-611812713062967954%40unknownmsgid
https://groups.google.com/d/msgid/elasticsearch/-611812713062967954%40unknownmsgid?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZH7S_hbh6RYmqpoN5qJX5M6MRdLEtH_WUbXTcbwrGNXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #13