Logstash pipeline output and elastic cloud

[alfredo.deluca (Alfredo)]

Hi all.
I have elastic cloud 7.10.2 and we are setting up logstash with a pipilne with input/filter/output. Also we are using helm to deploy.
Of course I have cloud.id and cloud.auth and not sure how to configure the outut section with those 2 field.

is it the below snippet correct:

output {
elasticsearch {
cloud_id => "MyID:KEY"
cloud_auth => "username:password"
}
}

Cheers

[aaron-nimocks]

Here are the docs and plugin config. Looks like you are doing it correctly.

cloud_id <label>:<cloud-id>

cloud_auth <username>:<password>

[alfredo.deluca (Alfredo)]

Thanks heaps Aaron. But when I run the logstash I got the following error:

Blockquote
warning: thread "Ruby-0-Thread-6: :1" terminated with exception (report_on_exception is true):
LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'ESURL:443/logstash'

Then another error:

Blockquote
[2021-02-09T09:06:22,973][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:363:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/common.rb:50:in `block in setup_after_successful_connection'"]}

Any idea?
Cheers

[aaron-nimocks]

You are getting your cloud ID from here?

Using `elastic:password? Try resetting password?

The URL ESURL:443/logstash which it tried to reach out looks wrong. Are you using GCP, Azure, or AWS?

[alfredo.deluca (Alfredo)]

Hi Aaron
ESURL:443/logstash is correct as I didn't want to expose my actual URL.

In the meantime I opened (all) the credentials for that user for the logstash and it's working fine but of course I'd rather give specific permission for that and all opened.

Is there a role/permission for logstash?
Cheers

[aaron-nimocks]

I just tested this out of the box and didn't change any settings/roles.

If you copy the ES endpoint and put it in the browser, enter user/pass, do you get a response? This is just testing authentication without Logstash.

[alfredo.deluca (Alfredo)]

all the endpoint elastic and kibana works perfectly. I deployed filebeat and metricbeat with no problems at all. With logstash I am using (like for file and metricbeat) a different user and not the user I use to login onto Kibana.

but I am having issue with the permission only for logstash as I have a pipeline with input/filter and output.

The rest works fine.

[aaron-nimocks]

Do you have any Traffic Filters set?

[alfredo.deluca (Alfredo)]

no traffic filters!

What I am looking for is the right permissions for a user created only for all the beats I ve been using (file,metric). For logstash I need special permission which I am still looking for.
The same user (not the cloud/root user) works fine with the metricbeat and filebeat... but not with logstash so I guess I need different permissions

Cheers

[aaron-nimocks]

Using your cloud user/password should have the correct permissions and works out of the box. If this doesn't work I would open an issue on GitHub.

Construct this value by following this format ":". Use your Cloud username for the first part. Use your Cloud password for the second part, which is given once in the Cloud UI when you create a cluster.

If you want to use a created user/roles instead of elastic then you need to configure those.

[alfredo.deluca (Alfredo)]

thanks Aaron. Unfortunately if I give only the logstash permission, I got the same error.