Logstash to elasticsearch service

Elastic Stack Logstash
1

Hi everyone,
I'm on my trial to test elasticcloud. But now I got problem to create pipeline from logstash to elasticcloud. Here is my logstash.conf output

output {
        stdout{codec=>rubydebug}
        elasticsearch
        {
            hosts=>["https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243"]
 index=>"testindex"
user=>elasticdeploymentcredentials
password=>elasticdeploymentcredentials
 }
}

But it always returning error as:

[WARN ] 2021-03-29 12:24:50.148 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:24:55.158 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:00.163 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 12:25:05.170 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 12:25:10.175 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}

following some steps from connecting logstash to elastic cloud

following instruction of warkolm, I test to curl it and it successfully returning

 "name" : "instance-0000000001",
  "cluster_name" : "<clusterid>",
  "cluster_uuid" : "<clusteruuid>",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

am I missing something?

2

Can you curl the cluster with those user credentials?

1 Like
3

Hi Warkolm, thank you for your reply.
[root@localhost testconfig]# curl https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.elastic-cloud.com:9243
returning

https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.elastic-cloud.com:9243
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
4

Then your credentials are incorrect, you may need to reset them.

1 Like
5 
  "name" : "instance-0000000001",
  "cluster_name" : "<clusterid>",
  "cluster_uuid" : "<clusteruuid>",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

reupdate my credentials. and update on logstash.conf password too. it returning

[WARN ] 2021-03-29 13:59:00.051 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 13:59:05.058 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}
[WARN ] 2021-03-29 13:59:10.065 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io: Name or service not known"}
[WARN ] 2021-03-29 13:59:15.071 [Ruby-0-Thread-9: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elasticdeploymentcredentials:elasticdeploymentcredentials@<clusterid>.asia-southeast1.gcp.cloud.es.io:9243/][Manticore::ResolutionFailure] <clusterid>.asia-southeast1.gcp.cloud.es.io"}

Please kindly help me.

6

Where do I can see the url of my elasticsearch service?

7

Check out Work with Elasticsearch | Elasticsearch Service Documentation | Elastic

1 Like
8

Dear walkolm, I've remake my configuration file and fill it with the right credentials, the curl also work. But the logstash conf still not working. Do you have any other solution? Thank you so much

9

Have you set up your username and password as string as shown here?

1 Like
10

Hi Christian, thank you for reply. I've add the user and password.

11

I'm so sorry, I've checked it out and I don't understand the instruction.

Before you begin

On the Overview page for your new cluster in the Elasticsearch Service Console, copy the Elasticsearch endpoint URL under Endpoints .

It only shows, Copy end point and copy cluster id. So I have to create my own.
End point could be shown like

https://i-o-optimized-deployment.es.asia-southeast1.gcp.elastic-cloud.com:9243

To create the hosts above, I Change i-o-optimized-deployment with cluster id manually. Then change domain elastic-cloud.com into cloud.es.io. I also remove es before region.
So my hosts become

https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243

still not working :confused:

12

You need to get the correct URL/endpoint from the console and make sure you are copying the Elasticsearch URL and not the Kibana one. I believe this should contain your cluster ID and that you should use it exactly as is.

1 Like
13

Dear @Christian_Dahlqvist , much thanks for your assistance. I've found the problem.
the url should be
"https://<clusterid>.es.asia-southeast1.gcp.elastic-cloud.com:9243"
instead of
https://<clusterid>.asia-southeast1.gcp.cloud.es.io:9243

and much thanks for @warkolm for your assistance.

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.