Discovery in EC2 VPC


(VegHead) #1

I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud
(VPC) using the cloud-aws plugin for Discovery. This works fine for me in
the normal AWS EC2 public cloud, but doesn't work in my VPC. For some
reason, ElasticSearch isn't discovering the other nodes. Strangely enough,
it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be part of
the cluster and definitely has the ElasticSearchDevVPC security group. It's
weird... it almost looks as if Reservation.getGroupNames() in the Java EC2
API is returning an empty array.

Has anyone seen this before?

-Sean

--


(matt-4) #2

Hi,

it may not be related but we had a similar issue recently that was down to
the ES cluster discovery using multicast by default. After switching to
unicast the discovery process completed.

On 20 August 2012 23:16, VegHead organicveggie@gmail.com wrote:

I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud
(VPC) using the cloud-aws plugin for Discovery. This works fine for me in
the normal AWS EC2 public cloud, but doesn't work in my VPC. For some
reason, ElasticSearch isn't discovering the other nodes. Strangely enough,
it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be part
of the cluster and definitely has the ElasticSearchDevVPC security group.
It's weird... it almost looks as if Reservation.getGroupNames() in the
Java EC2 API is returning an empty array.

Has anyone seen this before?

-Sean

--

--


(VegHead) #3

Thanks, but I'm definitely seeing a different problem. Multicast is
definitely turned off and I'm running several similar clusters in the
Amazon public cloud. It's only the cluster running the VPC that's having
trouble.

Discovery completes correctly. It's simply that other ElasticSearch
instances in the cluster are not getting found because they appear to be
displaying empty security groups.

-Sean

On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote:

Hi,

it may not be related but we had a similar issue recently that was down to
the ES cluster discovery using multicast by default. After switching to
unicast the discovery process completed.

On 20 August 2012 23:16, VegHead <organi...@gmail.com <javascript:>>wrote:

I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud
(VPC) using the cloud-aws plugin for Discovery. This works fine for me in
the normal AWS EC2 public cloud, but doesn't work in my VPC. For some
reason, ElasticSearch isn't discovering the other nodes. Strangely enough,
it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle Man]
filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be part
of the cluster and definitely has the ElasticSearchDevVPC security group.
It's weird... it almost looks as if Reservation.getGroupNames() in the
Java EC2 API is returning an empty array.

Has anyone seen this before?

-Sean

--


(Shay Banon) #4

Never had experience with EC2 VPC, the question is if the security groups applied are also applied to the VPC instances? I assume those instances are being listed and return when using the describe instances API (which ES uses)?

On Aug 22, 2012, at 5:10 PM, VegHead organicveggie@gmail.com wrote:

Thanks, but I'm definitely seeing a different problem. Multicast is definitely turned off and I'm running several similar clusters in the Amazon public cloud. It's only the cluster running the VPC that's having trouble.

Discovery completes correctly. It's simply that other ElasticSearch instances in the cluster are not getting found because they appear to be displaying empty security groups.

-Sean

On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote:
Hi,

it may not be related but we had a similar issue recently that was down to the ES cluster discovery using multicast by default. After switching to unicast the discovery process completed.

On 20 August 2012 23:16, VegHead organi...@gmail.com wrote:
I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud (VPC) using the cloud-aws plugin for Discovery. This works fine for me in the normal AWS EC2 public cloud, but doesn't work in my VPC. For some reason, ElasticSearch isn't discovering the other nodes. Strangely enough, it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle Man] filtering out reservation r-04754a60 based on groups [], not part of [ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle Man] filtering out reservation r-04754a60 based on groups [], not part of [ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle Man] filtering out reservation r-04754a60 based on groups [], not part of [ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be part of the cluster and definitely has the ElasticSearchDevVPC security group. It's weird... it almost looks as if Reservation.getGroupNames() in the Java EC2 API is returning an empty array.

Has anyone seen this before?

-Sean

--

--


(VegHead) #5

The instances listed when using the describe instances API. And I can vouch
that I see reservation numbers in the ES output that correspond to
instances in the VPC.

Security Groups do apply to VPC instances, but.... (you know there was a
"but" coming somewhere) it's a bit confusing. You have to explicitly define
Security Groups for VPCs, separately from the Security Groups you define
for the normal, public cloud. If you call the describeSecurityGroups API,
you get back the list of all security groups for the public cloud and all
VPCs. What makes this a bit wonky is that you can define a security group
with the same name in both the public cloud and in a VPC, but they're
actually two different security groups with two different ids.

For example, we have instances in the public cloud and instances in a VPC.
And we have security groups in both the public cloud and the vpc. In our
case, we have two security groups with the same name:

  • sg-44f6072c => "ElasticSearchDev", public cloud
  • sg-712dca1e => "ElasticSearchDev", vpc

If you iterate over the list of security groups returned by the
describeSecurityGroups API, you will see both security groups. However, you
can call getVpcId() on the security group. That returns null if it's not
associated with a VPC or the vpc id if it is associated with a VPC. You can
tell if an instance is within a VPC by calling getVpcId() on the Instance
object. Or you can look at the user meta-data and see if it has a
public-hostname (vpc instances do not have a public hostname or public ip).

Admittedly, that explanation is getting a little far off topic... because,
in theory, Reservation.getGroupNames() in the Java EC2 API should return
the security group names, regardless of whether the instance is in the
public cloud or a vpc.

I haven't tested this yet, but maybe if you go to the instances themselves
(Reservation.getInstances) and then call Instance.getSecurityGroups()
you'll get the right data back. Not sure. I'll try to open a ticket with
Amazon to clarify.

-Sean

On Wednesday, August 22, 2012 11:23:39 AM UTC-5, kimchy wrote:

Never had experience with EC2 VPC, the question is if the security groups
applied are also applied to the VPC instances? I assume those instances are
being listed and return when using the describe instances API (which ES
uses)?

On Aug 22, 2012, at 5:10 PM, VegHead <organi...@gmail.com <javascript:>>
wrote:

Thanks, but I'm definitely seeing a different problem. Multicast is
definitely turned off and I'm running several similar clusters in the
Amazon public cloud. It's only the cluster running the VPC that's having
trouble.

Discovery completes correctly. It's simply that other ElasticSearch
instances in the cluster are not getting found because they appear to be
displaying empty security groups.

-Sean

On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote:

Hi,

it may not be related but we had a similar issue recently that was down
to the ES cluster discovery using multicast by default. After switching to
unicast the discovery process completed.

On 20 August 2012 23:16, VegHead organi...@gmail.com wrote:

I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud
(VPC) using the cloud-aws plugin for Discovery. This works fine for me in
the normal AWS EC2 public cloud, but doesn't work in my VPC. For some
reason, ElasticSearch isn't discovering the other nodes. Strangely enough,
it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be
part of the cluster and definitely has the ElasticSearchDevVPC security
group. It's weird... it almost looks as if Reservation.getGroupNames()in the Java EC2 API is returning an empty array.

Has anyone seen this before?

-Sean

--


(VegHead) #6

Okay, the official word from Amazon is that ElasticSearch is not doing the
right thing. :slight_smile:

Reservation.getGroupNames() does not work for VPC security groups. Instead,
we're supposed to call Reservation.getInstances(), iterate over the
instances and then call Instance.getSecurityGroups().

I'll fix it and submit a pull request.

-Sean

On Wednesday, August 22, 2012 7:55:08 PM UTC-7, VegHead wrote:

The instances listed when using the describe instances API. And I can
vouch that I see reservation numbers in the ES output that correspond to
instances in the VPC.

Security Groups do apply to VPC instances, but.... (you know there was a
"but" coming somewhere) it's a bit confusing. You have to explicitly define
Security Groups for VPCs, separately from the Security Groups you define
for the normal, public cloud. If you call the describeSecurityGroups API,
you get back the list of all security groups for the public cloud and
all VPCs. What makes this a bit wonky is that you can define a security
group with the same name in both the public cloud and in a VPC, but they're
actually two different security groups with two different ids.

For example, we have instances in the public cloud and instances in a VPC.
And we have security groups in both the public cloud and the vpc. In our
case, we have two security groups with the same name:

  • sg-44f6072c => "ElasticSearchDev", public cloud
  • sg-712dca1e => "ElasticSearchDev", vpc

If you iterate over the list of security groups returned by the
describeSecurityGroups API, you will see both security groups. However, you
can call getVpcId() on the security group. That returns null if it's not
associated with a VPC or the vpc id if it is associated with a VPC. You can
tell if an instance is within a VPC by calling getVpcId() on the Instance
object. Or you can look at the user meta-data and see if it has a
public-hostname (vpc instances do not have a public hostname or public ip).

Admittedly, that explanation is getting a little far off topic... because,
in theory, Reservation.getGroupNames() in the Java EC2 API should return
the security group names, regardless of whether the instance is in the
public cloud or a vpc.

I haven't tested this yet, but maybe if you go to the instances themselves
(Reservation.getInstances) and then call Instance.getSecurityGroups()
you'll get the right data back. Not sure. I'll try to open a ticket with
Amazon to clarify.

-Sean

On Wednesday, August 22, 2012 11:23:39 AM UTC-5, kimchy wrote:

Never had experience with EC2 VPC, the question is if the security groups
applied are also applied to the VPC instances? I assume those instances are
being listed and return when using the describe instances API (which ES
uses)?

On Aug 22, 2012, at 5:10 PM, VegHead organi...@gmail.com wrote:

Thanks, but I'm definitely seeing a different problem. Multicast is
definitely turned off and I'm running several similar clusters in the
Amazon public cloud. It's only the cluster running the VPC that's having
trouble.

Discovery completes correctly. It's simply that other ElasticSearch
instances in the cluster are not getting found because they appear to be
displaying empty security groups.

-Sean

On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote:

Hi,

it may not be related but we had a similar issue recently that was down
to the ES cluster discovery using multicast by default. After switching to
unicast the discovery process completed.

On 20 August 2012 23:16, VegHead organi...@gmail.com wrote:

I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud
(VPC) using the cloud-aws plugin for Discovery. This works fine for me in
the normal AWS EC2 public cloud, but doesn't work in my VPC. For some
reason, ElasticSearch isn't discovering the other nodes. Strangely enough,
it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be
part of the cluster and definitely has the ElasticSearchDevVPC security
group. It's weird... it almost looks as if Reservation.getGroupNames()in the Java EC2 API is returning an empty array.

Has anyone seen this before?

-Sean

--


(VegHead) #7

Code committed and pull request sent.

-Sean

On Friday, August 24, 2012 8:42:36 AM UTC-7, VegHead wrote:

Okay, the official word from Amazon is that ElasticSearch is not doing the
right thing. :slight_smile:

Reservation.getGroupNames() does not work for VPC security groups.
Instead, we're supposed to call Reservation.getInstances(), iterate over
the instances and then call Instance.getSecurityGroups().

I'll fix it and submit a pull request.

-Sean

On Wednesday, August 22, 2012 7:55:08 PM UTC-7, VegHead wrote:

The instances listed when using the describe instances API. And I can
vouch that I see reservation numbers in the ES output that correspond to
instances in the VPC.

Security Groups do apply to VPC instances, but.... (you know there was a
"but" coming somewhere) it's a bit confusing. You have to explicitly define
Security Groups for VPCs, separately from the Security Groups you define
for the normal, public cloud. If you call the describeSecurityGroups API,
you get back the list of all security groups for the public cloud and
all VPCs. What makes this a bit wonky is that you can define a security
group with the same name in both the public cloud and in a VPC, but they're
actually two different security groups with two different ids.

For example, we have instances in the public cloud and instances in a
VPC. And we have security groups in both the public cloud and the vpc. In
our case, we have two security groups with the same name:

  • sg-44f6072c => "ElasticSearchDev", public cloud
  • sg-712dca1e => "ElasticSearchDev", vpc

If you iterate over the list of security groups returned by the
describeSecurityGroups API, you will see both security groups. However, you
can call getVpcId() on the security group. That returns null if it's not
associated with a VPC or the vpc id if it is associated with a VPC. You can
tell if an instance is within a VPC by calling getVpcId() on the
Instance object. Or you can look at the user meta-data and see if it has a
public-hostname (vpc instances do not have a public hostname or public ip).

Admittedly, that explanation is getting a little far off topic...
because, in theory, Reservation.getGroupNames() in the Java EC2 API
should return the security group names, regardless of whether the instance
is in the public cloud or a vpc.

I haven't tested this yet, but maybe if you go to the instances
themselves (Reservation.getInstances) and then call
Instance.getSecurityGroups() you'll get the right data back. Not sure. I'll
try to open a ticket with Amazon to clarify.

-Sean

On Wednesday, August 22, 2012 11:23:39 AM UTC-5, kimchy wrote:

Never had experience with EC2 VPC, the question is if the security
groups applied are also applied to the VPC instances? I assume those
instances are being listed and return when using the describe instances API
(which ES uses)?

On Aug 22, 2012, at 5:10 PM, VegHead organi...@gmail.com wrote:

Thanks, but I'm definitely seeing a different problem. Multicast is
definitely turned off and I'm running several similar clusters in the
Amazon public cloud. It's only the cluster running the VPC that's having
trouble.

Discovery completes correctly. It's simply that other ElasticSearch
instances in the cluster are not getting found because they appear to be
displaying empty security groups.

-Sean

On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote:

Hi,

it may not be related but we had a similar issue recently that was down
to the ES cluster discovery using multicast by default. After switching to
unicast the discovery process completed.

On 20 August 2012 23:16, VegHead organi...@gmail.com wrote:

I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud
(VPC) using the cloud-aws plugin for Discovery. This works fine for me in
the normal AWS EC2 public cloud, but doesn't work in my VPC. For some
reason, ElasticSearch isn't discovering the other nodes. Strangely enough,
it looks like it's not retrieving the group names correctly. For example:

[2012-08-20 16:28:17,160][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:19,193][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]
[2012-08-20 16:28:21,196][TRACE][discovery.ec2 ] [Miracle
Man] filtering out reservation r-04754a60 based on groups [], not part of
[ElasticSearchDevVPC]

In this case, reservation r-04754a60 is for a node that should be
part of the cluster and definitely has the ElasticSearchDevVPC security
group. It's weird... it almost looks as if Reservation.getGroupNames()in the Java EC2 API is returning an empty array.

Has anyone seen this before?

-Sean

--


(system) #8