# Logstash pipeline output and elastic cloud

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659>\
**Category:** Logstash\
**Created:** [February 8, 2021, 10:05pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659 "2021-02-08T22:05:47Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\
**Post date:** [February 8, 2021, 10:05pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/1 "2021-02-08T22:05:47Z")

</div>

Hi all.  
I have elastic cloud 7.10.2 and we are setting up logstash with a pipilne with input/filter/output. Also we are using helm to deploy.  
Of course I have cloud.id and cloud.auth and not sure how to configure the outut section with those 2 field.

is it the below snippet correct:

output {  
elasticsearch {  
cloud\_id =\> "MyID:KEY"  
cloud\_auth =\> "username:password"  
}  
}

Cheers

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [February 8, 2021, 10:21pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/2 "2021-02-08T22:21:54Z")

</div>

Here are the [docs](https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html) and [plugin config](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-cloud_auth). Looks like you are doing it correctly.

**cloud\_id** `<label>:<cloud-id>`

**cloud\_auth** `<username>:<password>`

---

<div class="post-metadata">

**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\
**Post date:** [February 9, 2021, 9:22am UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/3 "2021-02-09T09:22:11Z")

</div>

Thanks heaps Aaron. But when I run the logstash I got the following error:

> Blockquote  
> warning: thread "Ruby-0-Thread-6: :1" terminated with exception (report\_on\_exception is true):  
> LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'ESURL:443/logstash'

Then another error:

> Blockquote  
> [2021-02-09T09:06:22,973][FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\<LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError\>, :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http\_client/manticore\_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform\_request\_to\_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with\_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http\_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:363:in `rollover\_alias\_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup\_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/common.rb:50:in `block in setup\_after\_successful\_connection'"]}

Any idea?  
Cheers

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [February 9, 2021, 1:48pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/4 "2021-02-09T13:48:11Z")

</div>

You are getting your cloud ID from here?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de68fe51708ab006c4bbe77b44808e24ae890d69.png)

Using `elastic:password? Try resetting password?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a958bd1a4e09b547af0b2a4f4bfbda0940a87ebe.png)

The URL `ESURL:443/logstash` which it tried to reach out looks wrong. Are you using GCP, Azure, or AWS?

---

<div class="post-metadata">

**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\
**Post date:** [February 9, 2021, 2:38pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/5 "2021-02-09T14:38:04Z")

</div>

Hi Aaron  
ESURL:443/logstash is correct as I didn't want to expose my actual URL.

In the meantime I opened (all) the credentials for that user for the logstash and it's working fine but of course I'd rather give specific permission for that and all opened.

Is there a role/permission for logstash?  
Cheers

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [February 9, 2021, 2:52pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/6 "2021-02-09T14:52:17Z")

</div>

I just tested this out of the box and didn't change any settings/roles.

If you copy the ES endpoint and put it in the browser, enter user/pass, do you get a response? This is just testing authentication without Logstash.

---

<div class="post-metadata">

**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\
**Post date:** [February 9, 2021, 3:12pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/7 "2021-02-09T15:12:57Z")

</div>

all the endpoint elastic and kibana works perfectly. I deployed filebeat and metricbeat with no problems at all. With logstash I am using (like for file and metricbeat) a different user and not the user I use to login onto Kibana.

but I am having issue with the permission only for logstash as I have a pipeline with input/filter and output.

The rest works fine.

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [February 9, 2021, 3:24pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/8 "2021-02-09T15:24:10Z")

</div>

Do you have any [Traffic Filters](https://www.elastic.co/guide/en/cloud/current/ec-traffic-filtering-deployment-configuration.html) set?

---

<div class="post-metadata">

**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\
**Post date:** [February 10, 2021, 9:45am UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/9 "2021-02-10T09:45:14Z")

</div>

no traffic filters!

What I am looking for is the right permissions for a user created only for all the beats I ve been using (file,metric). For logstash I need special permission which I am still looking for.  
The same user (not the cloud/root user) works fine with the metricbeat and filebeat... but not with logstash so I guess I need different permissions

Cheers

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [February 10, 2021, 12:49pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/10 "2021-02-10T12:49:11Z")

</div>

Using your [cloud user/password](https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#cloud-auth) should have the correct permissions and works out of the box. If this doesn't work I would open an issue on GitHub.

> Construct this value by following this format ":". Use your Cloud username for the first part. Use your Cloud password for the second part, which is given once in the Cloud UI when you create a cluster.

If you want to use a created user/roles instead of `elastic` then you need to [configure](https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html#cloud-id-mgmt) those.

---

<div class="post-metadata">

**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\
**Post date:** [February 16, 2021, 12:17pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/11 "2021-02-16T12:17:29Z")

</div>

thanks Aaron. Unfortunately if I give only the logstash permission, I got the same error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2021, 12:17pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-and-elastic-cloud/263659/12 "2021-03-16T12:17:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.