Logstash to elasticsearch service

charles97 · March 29, 2021, 5:29am

Hi everyone,
I'm on my trial to test elasticcloud. But now I got problem to create pipeline from logstash to elasticcloud. Here is my logstash.conf output

output {
        stdout{codec=>rubydebug}
        elasticsearch
        {
            hosts=>["https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243"]
 index=>"testindex"
user=>elasticdeploymentcredentials
password=>elasticdeploymentcredentials
 }
}

But it always returning error as:

[WARN ] 2021-03-29 12:24:50.148 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:24:55.158 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:00.163 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:25:05.170 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:10.175 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}

following some steps from connecting logstash to elastic cloud

following instruction of warkolm, I test to curl it and it successfully returning

 "name" : "instance-0000000001",
  "cluster_name" : "<clusterid>",
  "cluster_uuid" : "<clusteruuid>",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

am I missing something?

warkolm · March 29, 2021, 6:28am

Can you curl the cluster with those user credentials?

charles97 · March 29, 2021, 6:46am

Hi Warkolm, thank you for your reply.
[root@localhost testconfig]# curl https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.elastic-cloud.com:9243
returning

https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.elastic-cloud.com:9243
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

warkolm · March 29, 2021, 6:58am

Then your credentials are incorrect, you may need to reset them.

charles97 · March 29, 2021, 7:02am

  "name" : "instance-0000000001",
  "cluster_name" : "<clusterid>",
  "cluster_uuid" : "<clusteruuid>",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

reupdate my credentials. and update on logstash.conf password too. it returning

[WARN ] 2021-03-29 13:59:00.051 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 13:59:05.058 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 13:59:10.065 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 13:59:15.071 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}

Please kindly help me.

charles97 · March 29, 2021, 7:25am

Where do I can see the url of my elasticsearch service?

warkolm · March 29, 2021, 8:45am

Check out Work with Elasticsearch | Elasticsearch Service Documentation | Elastic

charles97 · March 29, 2021, 8:49am

Dear walkolm, I've remake my configuration file and fill it with the right credentials, the curl also work. But the logstash conf still not working. Do you have any other solution? Thank you so much

Christian_Dahlqvist · March 29, 2021, 8:53am

Have you set up your username and password as string as shown here?

charles97 · March 29, 2021, 9:05am

Hi Christian, thank you for reply. I've add the user and password.

charles97 · March 29, 2021, 10:32am

I'm so sorry, I've checked it out and I don't understand the instruction.

Before you begin

On the Overview page for your new cluster in the Elasticsearch Service Console, copy the Elasticsearch endpoint URL under Endpoints .

It only shows, Copy end point and copy cluster id. So I have to create my own.
End point could be shown like

https://i-o-optimized-deployment.es.asia-southeast1.gcp.elastic-cloud.com:9243

To create the hosts above, I Change i-o-optimized-deployment with cluster id manually. Then change domain elastic-cloud.com into cloud.es.io. I also remove es before region.
So my hosts become

https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243

still not working

Christian_Dahlqvist · March 29, 2021, 11:06am

You need to get the correct URL/endpoint from the console and make sure you are copying the Elasticsearch URL and not the Kibana one. I believe this should contain your cluster ID and that you should use it exactly as is.

charles97 · March 30, 2021, 2:50am

Dear @Christian_Dahlqvist , much thanks for your assistance. I've found the problem.
the url should be
"https://<clusterid>.es.asia-southeast1.gcp.elastic-cloud.com:9243"
instead of
https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243

and much thanks for @warkolm for your assistance.

system · April 27, 2021, 2:51am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.