Azure Plugin - Error 403 can not get list of azure nodes

Elastic Stack Elasticsearch
1

Hi,

I've deployed a two nodes ElasticSearch cluster on Windows Azure. My setup
is the following :

  • I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe,
    other versions gave me trouble) to generate the SSH key, certificate and
    pkcs12 keystore
  • the Azure plugin (2.2.0) is installed on both nodes and defined as
    mandatory in elasticsearch.yml
  • the VMs run Ubuntu 12.04 (the exact image id is
    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB
    )

When I start the cluster I have the split brain syndrome, each node elects
itself as master and fails to see the other one. I configured the discovery
log level to TRACE to get more detailed information, and there is the
following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1]
can not get list of azure nodes: Server returned HTTP response code: 403
for URL:
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as
master.

I've attached the logs from both my nodes, as well as the
elasticsearch.yml config file (which only differs by setting a distinct
node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any
help would be much appreciated.

Best regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hey Nicolas,

The 403 status code from azure basically means that your credentials are incorrects.
It means to me that your certificate is either invalid in /home/elasticsearch/azurekeystore.pkcs12

You could try

curl --cert azure-cert.pem --key azure-pk.pem -H "x-ms-version: 2013-03-01" -H "Content-Type: application/json" "https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true"

And see if it works.

If not, I think

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 26 mai 2014 à 23:26:01, Nicolas Giraud (nicosensei@gmail.com) a écrit:

Hi,

I've deployed a two nodes ElasticSearch cluster on Windows Azure. My setup is the following :
I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe, other versions gave me trouble) to generate the SSH key, certificate and pkcs12 keystore
the Azure plugin (2.2.0) is installed on both nodes and defined as mandatory in elasticsearch.yml
the VMs run Ubuntu 12.04 (the exact image id is b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB)
When I start the cluster I have the split brain syndrome, each node elects itself as master and fails to see the other one. I configured the discovery log level to TRACE to get more detailed information, and there is the following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1] can not get list of azure nodes: Server returned HTTP response code: 403 for URL: https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as master.

I've attached the logs from both my nodes, as well as the elasticsearch.yml config file (which only differs by setting a distinct node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any help would be much appreciated.

Best regards,

Nicolas

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53845ded.275ac794.1e56%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

3

Ok, I'll try that as soon as I can. One (maybe dumb) question meanwhile, do
the credentials provided when creating the certificate (I followed these
steps :
http://azure.microsoft.com/en-us/documentation/articles/linux-use-ssh-key/)
need to match the Azure account credentials (email / password) ?

On Tuesday, May 27, 2014 11:42:13 AM UTC+2, David Pilato wrote:

Hey Nicolas,

The 403 status code from azure basically means that your credentials are
incorrects.
It means to me that your certificate is either invalid
in /home/elasticsearch/azurekeystore.pkcs12

You could try

curl --cert azure-cert.pem --key azure-pk.pem -H "x-ms-version:
2013-03-01" -H "Content-Type: application/json" "
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true
"

And see if it works.

If not, I think

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 26 mai 2014 à 23:26:01, Nicolas Giraud (nicos...@gmail.com<javascript:>)
a écrit:

Hi,

I've deployed a two nodes Elasticsearch cluster on Windows Azure. My setup
is the following :

  • I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe,
    other versions gave me trouble) to generate the SSH key, certificate and
    pkcs12 keystore
  • the Azure plugin (2.2.0) is installed on both nodes and defined as
    mandatory in elasticsearch.yml
  • the VMs run Ubuntu 12.04 (the exact image id is
    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB
    )

When I start the cluster I have the split brain syndrome, each node
elects itself as master and fails to see the other one. I configured the
discovery log level to TRACE to get more detailed information, and there is
the following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1]
can not get list of azure nodes: Server returned HTTP response code: 403
for URL:
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as
master.

I've attached the logs from both my nodes, as well as the
elasticsearch.yml config file (which only differs by setting a distinct
node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any
help would be much appreciated.

Best regards,

Nicolas

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6fe8613-6929-4b01-a9d2-cc6bb921f587%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

No they don't have to match.
The certificate have to be uploaded to Azure platform and that's all. Whatever your email address is.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 27 mai 2014 à 12:17:16, Nicolas Giraud (nicosensei@gmail.com) a écrit:

Ok, I'll try that as soon as I can. One (maybe dumb) question meanwhile, do the credentials provided when creating the certificate (I followed these steps : http://azure.microsoft.com/en-us/documentation/articles/linux-use-ssh-key/) need to match the Azure account credentials (email / password) ?

On Tuesday, May 27, 2014 11:42:13 AM UTC+2, David Pilato wrote:
Hey Nicolas,

The 403 status code from azure basically means that your credentials are incorrects.
It means to me that your certificate is either invalid in /home/elasticsearch/azurekeystore.pkcs12

You could try

curl --cert azure-cert.pem --key azure-pk.pem -H "x-ms-version: 2013-03-01" -H "Content-Type: application/json" "https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true"

And see if it works.

If not, I think

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 26 mai 2014 à 23:26:01, Nicolas Giraud (nicos...@gmail.com) a écrit:

Hi,

I've deployed a two nodes ElasticSearch cluster on Windows Azure. My setup is the following :
I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe, other versions gave me trouble) to generate the SSH key, certificate and pkcs12 keystore
the Azure plugin (2.2.0) is installed on both nodes and defined as mandatory in elasticsearch.yml
the VMs run Ubuntu 12.04 (the exact image id is b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB)
When I start the cluster I have the split brain syndrome, each node elects itself as master and fails to see the other one. I configured the discovery log level to TRACE to get more detailed information, and there is the following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1] can not get list of azure nodes: Server returned HTTP response code: 403 for URL: https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as master.

I've attached the logs from both my nodes, as well as the elasticsearch.yml config file (which only differs by setting a distinct node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any help would be much appreciated.

Best regards,

Nicolas

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6fe8613-6929-4b01-a9d2-cc6bb921f587%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53847e81.1f48eaa1.1e56%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

5

I tried as you suggested :

curl --cert azure-certificate.pem --key azure-pk.pem -H "x-ms-version:
2013-03-01" -H "Content-Type: application/json"
"https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true"

and got the same error as with ES :

ForbiddenErrorThe
server failed to authenticate the request. Verify that the certificate is
valid and is associated with this subscription.

I'm using the Linux azure-cli and bash shells to deploy my cluster. I built
OpenSSL 1.0.1c from source, and here are the commands I use to generate the
certificate, private key and Java keystore (pretty much copy pasted from
the blog article) :

OPENSSL_BIN=/usr/local/ssl/bin/openssl
$OPENSSL_BIN req -x509 -nodes -days 365 -newkey rsa:2048 -keyout $PRIVKEY
-out $CERT
chmod 600 $PRIVKEY
$OPENSSL_BIN x509 -outform der -in $CERT -out $CERT_DER

Generate Java keystore

$OPENSSL_BIN pkcs8 -topk8 -nocrypt -in $PRIVKEY -inform PEM -out
azure-pk.pem -outform PEM
cat $CERT azure-pk.pem > azure.pem.txt
$OPENSSL_BIN pkcs12 -export -in azure.pem.txt -out $KEYSTORE -name azure
-noiter -nomaciter

The certificate has been uploaded when I created the initial VM and the
cloud service was subsequently created:

CERT=azure-certificate.pem
SERVICE=elasticpoc
HOST=$SERVICE.cloudapp.net
USER=elasticsearch
VM_PWD=esAzure1!!
IMG=ubuntu-java7-elasticsearch
VM_SIZE=extralarge
OS_IMAGE=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB

azure vm create $HOST $OS_IMAGE
--vm-name $IMG
--location "West Europe"
--vm-size $VM_SIZE
--ssh 22
--ssh-cert ssl/$CERT
$USER $VM_PWD

VM_PWD is the same as the certificate password.

The keystore is copied over SSH to the final VMs once the ne above has been
set up and captured.

I don't understand what's happening here ...

On Tuesday, May 27, 2014 2:01:13 PM UTC+2, David Pilato wrote:

No they don't have to match.
The certificate have to be uploaded to Azure platform and that's all.
Whatever your email address is.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 27 mai 2014 à 12:17:16, Nicolas Giraud (nicos...@gmail.com<javascript:>)
a écrit:

Ok, I'll try that as soon as I can. One (maybe dumb) question meanwhile,
do the credentials provided when creating the certificate (I followed these
steps :
http://azure.microsoft.com/en-us/documentation/articles/linux-use-ssh-key/)
need to match the Azure account credentials (email / password) ?

On Tuesday, May 27, 2014 11:42:13 AM UTC+2, David Pilato wrote:

Hey Nicolas,

The 403 status code from azure basically means that your credentials are
incorrects.
It means to me that your certificate is either invalid
in /home/elasticsearch/azurekeystore.pkcs12

You could try

curl --cert azure-cert.pem --key azure-pk.pem -H "x-ms-version:
2013-03-01" -H "Content-Type: application/json" "
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true
"

And see if it works.

If not, I think

 --

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 26 mai 2014 à 23:26:01, Nicolas Giraud (nicos...@gmail.com) a écrit:

Hi,

I've deployed a two nodes Elasticsearch cluster on Windows Azure. My
setup is the following :

  • I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe,
    other versions gave me trouble) to generate the SSH key, certificate and
    pkcs12 keystore
  • the Azure plugin (2.2.0) is installed on both nodes and defined as
    mandatory in elasticsearch.yml
  • the VMs run Ubuntu 12.04 (the exact image id is
    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB
    )

When I start the cluster I have the split brain syndrome, each node
elects itself as master and fails to see the other one. I configured the
discovery log level to TRACE to get more detailed information, and there is
the following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1]
can not get list of azure nodes: Server returned HTTP response code: 403
for URL:
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as
master.

I've attached the logs from both my nodes, as well as the
elasticsearch.yml config file (which only differs by setting a
distinct node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any
help would be much appreciated.

Best regards,

Nicolas

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/39ed88e3-c30c-428a-a65f-c76cfbf99ec2%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b6fe8613-6929-4b01-a9d2-cc6bb921f587%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b6fe8613-6929-4b01-a9d2-cc6bb921f587%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3cadf1d6-54cc-4293-a578-0d4424de6bd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

I found some relevant info
here: How to deploy to Azure with powershell? - Stack Overflow

The curl command now works. I'm currently redeploying my service, fingers
crossed!

On Monday, May 26, 2014 11:25:58 PM UTC+2, Nicolas Giraud wrote:

Hi,

I've deployed a two nodes Elasticsearch cluster on Windows Azure. My setup
is the following :

  • I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe,
    other versions gave me trouble) to generate the SSH key, certificate and
    pkcs12 keystore
  • the Azure plugin (2.2.0) is installed on both nodes and defined as
    mandatory in elasticsearch.yml
  • the VMs run Ubuntu 12.04 (the exact image id is
    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB
    )

When I start the cluster I have the split brain syndrome, each node
elects itself as master and fails to see the other one. I configured the
discovery log level to TRACE to get more detailed information, and there is
the following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1]
can not get list of azure nodes: Server returned HTTP response code: 403
for URL:
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as
master.

I've attached the logs from both my nodes, as well as the
elasticsearch.yml config file (which only differs by setting a distinct
node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any
help would be much appreciated.

Best regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bdc6277a-b81f-4556-902d-832c3032ba3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

I confirm that this works. I simply needed to upload my PEM certificate to
Azure under Settings/Management Certificates.

Simply uploading it with the cloud service is not enough ... dumb mistake
in the end :wink:

On Tuesday, May 27, 2014 11:15:10 PM UTC+2, Nicolas Giraud wrote:

I found some relevant info here:
How to deploy to Azure with powershell? - Stack Overflow

The curl command now works. I'm currently redeploying my service, fingers
crossed!

On Monday, May 26, 2014 11:25:58 PM UTC+2, Nicolas Giraud wrote:

Hi,

I've deployed a two nodes Elasticsearch cluster on Windows Azure. My
setup is the following :

  • I use OpenSSL 1.0.1c (as recommended on the plugin's GitHub pahe,
    other versions gave me trouble) to generate the SSH key, certificate and
    pkcs12 keystore
  • the Azure plugin (2.2.0) is installed on both nodes and defined as
    mandatory in elasticsearch.yml
  • the VMs run Ubuntu 12.04 (the exact image id is
    b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140514-en-us-30GB
    )

When I start the cluster I have the split brain syndrome, each node
elects itself as master and fails to see the other one. I configured the
discovery log level to TRACE to get more detailed information, and there is
the following error message :

[2014-05-26 17:46:21,285][WARN ][cloud.azure ] [elasticpoc1]
can not get list of azure nodes: Server returned HTTP response code: 403
for URL:
https://management.core.windows.net/1d4c95fb-d9f1-4594-af6b-bfd3941f1c64/services/hostedservices/elasticpoc?embed-detail=true

This error appears 3 times in the log before the local node is elected as
master.

I've attached the logs from both my nodes, as well as the
elasticsearch.yml config file (which only differs by setting a
distinct node name between the 2 nodes).

I'm pretty clueless as to how I should proceed to get this right, so any
help would be much appreciated.

Best regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9ef6c2bd-30a1-450b-ad41-0330a5815180%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.