# Discovery in EC2 VPC **URL:** \ **Category:** Elasticsearch\ **Created:** [August 20, 2012, 10:16pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793 "2012-08-20T22:16:05Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [August 20, 2012, 10:16pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/1 "2012-08-20T22:16:05Z")
I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud (VPC) using the cloud-aws plugin for Discovery. This works fine for me in the normal AWS EC2 public cloud, but doesn't work in my VPC. For some reason, ElasticSearch isn't discovering the other nodes. Strangely enough, it looks like it's not retrieving the group names correctly. For example: [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle Man] filtering out reservation r-04754a60 based on groups [], not part of [ElasticSearchDevVPC] [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle Man] filtering out reservation r-04754a60 based on groups [], not part of [ElasticSearchDevVPC] [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle Man] filtering out reservation r-04754a60 based on groups [], not part of [ElasticSearchDevVPC] In this case, reservation r-04754a60 is for a node that _should_ be part of the cluster and definitely has the ElasticSearchDevVPC security group. It's weird... it almost looks as if Reservation.getGroupNames() in the Java EC2 API is returning an empty array. Has anyone seen this before? -Sean -- ---
**Author:** ![matt\_4](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@matt\_4](https://discuss.elastic.co/u/matt_4)\ **Post date:** [August 21, 2012, 10:06pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/2 "2012-08-21T22:06:00Z")
Hi, it may not be related but we had a similar issue recently that was down to the ES cluster discovery using multicast by default. After switching to unicast the discovery process completed. On 20 August 2012 23:16, VegHead [organicveggie@gmail.com](mailto:organicveggie@gmail.com) wrote: > I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud > (VPC) using the cloud-aws plugin for Discovery. This works fine for me in > the normal AWS EC2 public cloud, but doesn't work in my VPC. For some > reason, Elasticsearch isn't discovering the other nodes. Strangely enough, > it looks like it's not retrieving the group names correctly. For example: > > [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle Man] > filtering out reservation r-04754a60 based on groups , not part of > [ElasticSearchDevVPC] > [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle Man] > filtering out reservation r-04754a60 based on groups , not part of > [ElasticSearchDevVPC] > [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle Man] > filtering out reservation r-04754a60 based on groups , not part of > [ElasticSearchDevVPC] > > In this case, reservation r-04754a60 is for a node that _should_ be part > of the cluster and definitely has the ElasticSearchDevVPC security group. > It's weird... it almost looks as if Reservation.getGroupNames() in the > Java EC2 API is returning an empty array. > > Has anyone seen this before? > > -Sean > > -- -- ---
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [August 22, 2012, 3:10pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/3 "2012-08-22T15:10:27Z")
Thanks, but I'm definitely seeing a different problem. Multicast is definitely turned off and I'm running several similar clusters in the Amazon public cloud. It's only the cluster running the VPC that's having trouble. Discovery completes correctly. It's simply that other Elasticsearch instances in the cluster are not getting found because they appear to be displaying empty security groups. -Sean On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote: > Hi, > > it may not be related but we had a similar issue recently that was down to > the ES cluster discovery using multicast by default. After switching to > unicast the discovery process completed. > > On 20 August 2012 23:16, VegHead \<[organi...@gmail.com](mailto:organi...@gmail.com) \\>wrote: > > > I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud > > (VPC) using the cloud-aws plugin for Discovery. This works fine for me in > > the normal AWS EC2 public cloud, but doesn't work in my VPC. For some > > reason, Elasticsearch isn't discovering the other nodes. Strangely enough, > > it looks like it's not retrieving the group names correctly. For example: > > > > [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle Man] > > filtering out reservation r-04754a60 based on groups , not part of > > [ElasticSearchDevVPC] > > [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle Man] > > filtering out reservation r-04754a60 based on groups , not part of > > [ElasticSearchDevVPC] > > [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle Man] > > filtering out reservation r-04754a60 based on groups , not part of > > [ElasticSearchDevVPC] > > > > In this case, reservation r-04754a60 is for a node that _should_ be part > > of the cluster and definitely has the ElasticSearchDevVPC security group. > > It's weird... it almost looks as if Reservation.getGroupNames() in the > > Java EC2 API is returning an empty array. > > > > Has anyone seen this before? > > > > -Sean -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 22, 2012, 4:23pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/4 "2012-08-22T16:23:39Z")
Never had experience with EC2 VPC, the question is if the security groups applied are also applied to the VPC instances? I assume those instances are being listed and return when using the describe instances API (which ES uses)? On Aug 22, 2012, at 5:10 PM, VegHead [organicveggie@gmail.com](mailto:organicveggie@gmail.com) wrote: > Thanks, but I'm definitely seeing a different problem. Multicast is definitely turned off and I'm running several similar clusters in the Amazon public cloud. It's only the cluster running the VPC that's having trouble. > > Discovery completes correctly. It's simply that other Elasticsearch instances in the cluster are not getting found because they appear to be displaying empty security groups. > > -Sean > > On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote: > Hi, > > it may not be related but we had a similar issue recently that was down to the ES cluster discovery using multicast by default. After switching to unicast the discovery process completed. > > On 20 August 2012 23:16, VegHead [organi...@gmail.com](mailto:organi...@gmail.com) wrote: > I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud (VPC) using the cloud-aws plugin for Discovery. This works fine for me in the normal AWS EC2 public cloud, but doesn't work in my VPC. For some reason, Elasticsearch isn't discovering the other nodes. Strangely enough, it looks like it's not retrieving the group names correctly. For example: > > [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle Man] filtering out reservation r-04754a60 based on groups , not part of [ElasticSearchDevVPC] > [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle Man] filtering out reservation r-04754a60 based on groups , not part of [ElasticSearchDevVPC] > [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle Man] filtering out reservation r-04754a60 based on groups , not part of [ElasticSearchDevVPC] > > In this case, reservation r-04754a60 is for a node that should be part of the cluster and definitely has the ElasticSearchDevVPC security group. It's weird... it almost looks as if Reservation.getGroupNames() in the Java EC2 API is returning an empty array. > > Has anyone seen this before? > > -Sean > > -- -- ---
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [August 23, 2012, 2:55am UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/5 "2012-08-23T02:55:08Z")
The instances listed when using the describe instances API. And I can vouch that I see reservation numbers in the ES output that correspond to instances in the VPC. Security Groups do apply to VPC instances, but.... (you know there was a "but" coming somewhere) it's a bit confusing. You have to explicitly define Security Groups for VPCs, separately from the Security Groups you define for the normal, public cloud. If you call the describeSecurityGroups API, you get back the list of _all_ security groups for the public cloud and all VPCs. What makes this a bit wonky is that you can define a security group with the same name in both the public cloud and in a VPC, but they're actually two different security groups with two different ids. For example, we have instances in the public cloud and instances in a VPC. And we have security groups in both the public cloud and the vpc. In our case, we have two security groups with the same name: - sg-44f6072c =\> "ElasticSearchDev", public cloud - sg-712dca1e =\> "ElasticSearchDev", vpc If you iterate over the list of security groups returned by the describeSecurityGroups API, you will see both security groups. However, you can call getVpcId() on the security group. That returns null if it's not associated with a VPC or the vpc id if it is associated with a VPC. You can tell if an instance is within a VPC by calling getVpcId() on the Instance object. Or you can look at the user meta-data and see if it has a public-hostname (vpc instances do not have a public hostname or public ip). Admittedly, that explanation is getting a little far off topic... because, in theory, Reservation.getGroupNames() in the Java EC2 API should return the security group names, regardless of whether the instance is in the public cloud or a vpc. I haven't tested this yet, but maybe if you go to the instances themselves (Reservation.getInstances) and then call Instance.getSecurityGroups() you'll get the right data back. Not sure. I'll try to open a ticket with Amazon to clarify. -Sean On Wednesday, August 22, 2012 11:23:39 AM UTC-5, kimchy wrote: > Never had experience with EC2 VPC, the question is if the security groups > applied are also applied to the VPC instances? I assume those instances are > being listed and return when using the describe instances API (which ES > uses)? > > On Aug 22, 2012, at 5:10 PM, VegHead \<[organi...@gmail.com](mailto:organi...@gmail.com) \\> > wrote: > > Thanks, but I'm definitely seeing a different problem. Multicast is > definitely turned off and I'm running several similar clusters in the > Amazon public cloud. It's only the cluster running the VPC that's having > trouble. > > Discovery completes correctly. It's simply that other Elasticsearch > instances in the cluster are not getting found because they appear to be > displaying empty security groups. > > -Sean > > On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote: > > > Hi, > > > > it may not be related but we had a similar issue recently that was down > > to the ES cluster discovery using multicast by default. After switching to > > unicast the discovery process completed. > > > > On 20 August 2012 23:16, VegHead [organi...@gmail.com](mailto:organi...@gmail.com) wrote: > > > > > I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud > > > (VPC) using the cloud-aws plugin for Discovery. This works fine for me in > > > the normal AWS EC2 public cloud, but doesn't work in my VPC. For some > > > reason, Elasticsearch isn't discovering the other nodes. Strangely enough, > > > it looks like it's not retrieving the group names correctly. For example: > > > > > > [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > [ElasticSearchDevVPC] > > > [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > [ElasticSearchDevVPC] > > > [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > [ElasticSearchDevVPC] > > > > > > In this case, reservation r-04754a60 is for a node that _should_ be > > > part of the cluster and definitely has the ElasticSearchDevVPC security > > > group. It's weird... it almost looks as if Reservation.getGroupNames()in the Java EC2 API is returning an empty array. > > > > > > Has anyone seen this before? > > > > > > -Sean -- ---
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [August 24, 2012, 3:42pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/6 "2012-08-24T15:42:36Z")
Okay, the official word from Amazon is that Elasticsearch is not doing the right thing. πŸ™‚ Reservation.getGroupNames() does not work for VPC security groups. Instead, we're supposed to call Reservation.getInstances(), iterate over the instances and then call Instance.getSecurityGroups(). I'll fix it and submit a pull request. -Sean On Wednesday, August 22, 2012 7:55:08 PM UTC-7, VegHead wrote: > The instances listed when using the describe instances API. And I can > vouch that I see reservation numbers in the ES output that correspond to > instances in the VPC. > > Security Groups do apply to VPC instances, but.... (you know there was a > "but" coming somewhere) it's a bit confusing. You have to explicitly define > Security Groups for VPCs, separately from the Security Groups you define > for the normal, public cloud. If you call the describeSecurityGroups API, > you get back the list of _all_ security groups for the public cloud and > all VPCs. What makes this a bit wonky is that you can define a security > group with the same name in both the public cloud and in a VPC, but they're > actually two different security groups with two different ids. > > For example, we have instances in the public cloud and instances in a VPC. > And we have security groups in both the public cloud and the vpc. In our > case, we have two security groups with the same name: > > - sg-44f6072c =\> "ElasticSearchDev", public cloud > - sg-712dca1e =\> "ElasticSearchDev", vpc > > If you iterate over the list of security groups returned by the > describeSecurityGroups API, you will see both security groups. However, you > can call getVpcId() on the security group. That returns null if it's not > associated with a VPC or the vpc id if it is associated with a VPC. You can > tell if an instance is within a VPC by calling getVpcId() on the Instance > object. Or you can look at the user meta-data and see if it has a > public-hostname (vpc instances do not have a public hostname or public ip). > > Admittedly, that explanation is getting a little far off topic... because, > in theory, Reservation.getGroupNames() in the Java EC2 API should return > the security group names, regardless of whether the instance is in the > public cloud or a vpc. > > I haven't tested this yet, but maybe if you go to the instances themselves > (Reservation.getInstances) and then call Instance.getSecurityGroups() > you'll get the right data back. Not sure. I'll try to open a ticket with > Amazon to clarify. > > -Sean > > On Wednesday, August 22, 2012 11:23:39 AM UTC-5, kimchy wrote: > > > Never had experience with EC2 VPC, the question is if the security groups > > applied are also applied to the VPC instances? I assume those instances are > > being listed and return when using the describe instances API (which ES > > uses)? > > > > On Aug 22, 2012, at 5:10 PM, VegHead [organi...@gmail.com](mailto:organi...@gmail.com) wrote: > > > > Thanks, but I'm definitely seeing a different problem. Multicast is > > definitely turned off and I'm running several similar clusters in the > > Amazon public cloud. It's only the cluster running the VPC that's having > > trouble. > > > > Discovery completes correctly. It's simply that other Elasticsearch > > instances in the cluster are not getting found because they appear to be > > displaying empty security groups. > > > > -Sean > > > > On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote: > > > > > Hi, > > > > > > it may not be related but we had a similar issue recently that was down > > > to the ES cluster discovery using multicast by default. After switching to > > > unicast the discovery process completed. > > > > > > On 20 August 2012 23:16, VegHead [organi...@gmail.com](mailto:organi...@gmail.com) wrote: > > > > > > > I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud > > > > (VPC) using the cloud-aws plugin for Discovery. This works fine for me in > > > > the normal AWS EC2 public cloud, but doesn't work in my VPC. For some > > > > reason, Elasticsearch isn't discovering the other nodes. Strangely enough, > > > > it looks like it's not retrieving the group names correctly. For example: > > > > > > > > [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle > > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > > [ElasticSearchDevVPC] > > > > [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle > > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > > [ElasticSearchDevVPC] > > > > [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle > > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > > [ElasticSearchDevVPC] > > > > > > > > In this case, reservation r-04754a60 is for a node that _should_ be > > > > part of the cluster and definitely has the ElasticSearchDevVPC security > > > > group. It's weird... it almost looks as if Reservation.getGroupNames()in the Java EC2 API is returning an empty array. > > > > > > > > Has anyone seen this before? > > > > > > > > -Sean -- ---
**Author:** ![VegHead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/veghead/32/2411_2.png) [@VegHead](https://discuss.elastic.co/u/VegHead)\ **Post date:** [August 24, 2012, 11:26pm UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/7 "2012-08-24T23:26:44Z")
Code committed and pull request sent. > > > Fixed EC2 discovery to correctly find security groups, regardless of whether the… instance is in the public cloud or within a VPC. -Sean On Friday, August 24, 2012 8:42:36 AM UTC-7, VegHead wrote: > Okay, the official word from Amazon is that Elasticsearch is not doing the > right thing. πŸ™‚ > > Reservation.getGroupNames() does not work for VPC security groups. > Instead, we're supposed to call Reservation.getInstances(), iterate over > the instances and then call Instance.getSecurityGroups(). > > I'll fix it and submit a pull request. > > -Sean > > On Wednesday, August 22, 2012 7:55:08 PM UTC-7, VegHead wrote: > > > The instances listed when using the describe instances API. And I can > > vouch that I see reservation numbers in the ES output that correspond to > > instances in the VPC. > > > > Security Groups do apply to VPC instances, but.... (you know there was a > > "but" coming somewhere) it's a bit confusing. You have to explicitly define > > Security Groups for VPCs, separately from the Security Groups you define > > for the normal, public cloud. If you call the describeSecurityGroups API, > > you get back the list of _all_ security groups for the public cloud and > > all VPCs. What makes this a bit wonky is that you can define a security > > group with the same name in both the public cloud and in a VPC, but they're > > actually two different security groups with two different ids. > > > > For example, we have instances in the public cloud and instances in a > > VPC. And we have security groups in both the public cloud and the vpc. In > > our case, we have two security groups with the same name: > > > > - sg-44f6072c =\> "ElasticSearchDev", public cloud > > - sg-712dca1e =\> "ElasticSearchDev", vpc > > > > If you iterate over the list of security groups returned by the > > describeSecurityGroups API, you will see both security groups. However, you > > can call getVpcId() on the security group. That returns null if it's not > > associated with a VPC or the vpc id if it is associated with a VPC. You can > > tell if an instance is within a VPC by calling getVpcId() on the > > Instance object. Or you can look at the user meta-data and see if it has a > > public-hostname (vpc instances do not have a public hostname or public ip). > > > > Admittedly, that explanation is getting a little far off topic... > > because, in theory, Reservation.getGroupNames() in the Java EC2 API > > should return the security group names, regardless of whether the instance > > is in the public cloud or a vpc. > > > > I haven't tested this yet, but maybe if you go to the instances > > themselves (Reservation.getInstances) and then call > > Instance.getSecurityGroups() you'll get the right data back. Not sure. I'll > > try to open a ticket with Amazon to clarify. > > > > -Sean > > > > On Wednesday, August 22, 2012 11:23:39 AM UTC-5, kimchy wrote: > > > > > Never had experience with EC2 VPC, the question is if the security > > > groups applied are also applied to the VPC instances? I assume those > > > instances are being listed and return when using the describe instances API > > > (which ES uses)? > > > > > > On Aug 22, 2012, at 5:10 PM, VegHead [organi...@gmail.com](mailto:organi...@gmail.com) wrote: > > > > > > Thanks, but I'm definitely seeing a different problem. Multicast is > > > definitely turned off and I'm running several similar clusters in the > > > Amazon public cloud. It's only the cluster running the VPC that's having > > > trouble. > > > > > > Discovery completes correctly. It's simply that other Elasticsearch > > > instances in the cluster are not getting found because they appear to be > > > displaying empty security groups. > > > > > > -Sean > > > > > > On Tuesday, August 21, 2012 5:06:00 PM UTC-5, Matt Prime wrote: > > > > > > > Hi, > > > > > > > > it may not be related but we had a similar issue recently that was down > > > > to the ES cluster discovery using multicast by default. After switching to > > > > unicast the discovery process completed. > > > > > > > > On 20 August 2012 23:16, VegHead [organi...@gmail.com](mailto:organi...@gmail.com) wrote: > > > > > > > > > I'm trying to launch a 10 node cluster in an EC2 Virtual Private Cloud > > > > > (VPC) using the cloud-aws plugin for Discovery. This works fine for me in > > > > > the normal AWS EC2 public cloud, but doesn't work in my VPC. For some > > > > > reason, Elasticsearch isn't discovering the other nodes. Strangely enough, > > > > > it looks like it's not retrieving the group names correctly. For example: > > > > > > > > > > [2012-08-20 16:28:17,160][TRACE][discovery.ec2] [Miracle > > > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > > > [ElasticSearchDevVPC] > > > > > [2012-08-20 16:28:19,193][TRACE][discovery.ec2] [Miracle > > > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > > > [ElasticSearchDevVPC] > > > > > [2012-08-20 16:28:21,196][TRACE][discovery.ec2] [Miracle > > > > > Man] filtering out reservation r-04754a60 based on groups , not part of > > > > > [ElasticSearchDevVPC] > > > > > > > > > > In this case, reservation r-04754a60 is for a node that _should_ be > > > > > part of the cluster and definitely has the ElasticSearchDevVPC security > > > > > group. It's weird... it almost looks as if Reservation.getGroupNames()in the Java EC2 API is returning an empty array. > > > > > > > > > > Has anyone seen this before? > > > > > > > > > > -Sean -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:15am UTC](https://discuss.elastic.co/t/discovery-in-ec2-vpc/8793/8 "2017-07-06T03:15:04Z")