Connecting Logstash to Elastic Cloud

[BezVeze (МаКи БезВезе)]

Hi there,

Can anyone help me around with connecting logstash to elastic cloud ive been struggling a little for a while...

let me tell you how my environment is:

I have google cloud compute engine machine that have filebeat installed on it and send a logfile to Logstash that is implemented as a container in a kubernetes cluster...

Now when i tell logstash to connect to elasticsearch that is also as a container in my kubernetes cluster it works ok, so the pipeline so far is good for my staging environment.

But the thing is for my production env i will use elastic cloud as a elasticsearch service and i need to setup logstash container to parse the output log files into elastic cloud cluster.

Im confused about two things,
ive alreade configured logstash pipeline like this:

input {
  beats {
    port => 5044
  }
}
 
filter {
doesent matter
}
 
output {
  elasticsearch { 
    hosts => ["elasticsearch:9200"] 
  }
}

but the output here i use is for my local elasticsearch in k8s,

Then again to use the elastic cloud i need to modify the logstash.yaml configuration file as it says here

https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html

im really confused here? do i need to specify output in my pipeline or i can just configure input and filter,

How this really works does someone have some example configuration file to share

Can someone help me around, Thanks in advance

[Christian_Dahlqvist]

Try something like this:

elasticsearch {
  hosts => ["https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.us-central1.gcp.cloud.es.io:9243"]
  user => "elastic"
  password => "password"
}

[BezVeze (МаКи БезВезе)]

Nope,

It doesent work,

Maybe you can point a way for debugging somehow,

And if this is the way to connect what is the deal of using Cloud ID?

[Christian_Dahlqvist]

As indicated by the docs you linked to, Cloud ID is used for Logstash Modules and Monitoring:

Cloud ID applies only when a Logstash module is enabled, otherwise specifying Cloud ID has no effect. Cloud ID applies to data that gets sent via the module, to runtime metrics sent via X-Pack monitoring, and to the endpoint used by X-Pack central management features of Logstash, unless explicit overrides to X-Pack settings are specified in logstash.yml.

The config I posted works for me, so I would expect it to work for you too once you have set the correct hosts string, username and password. Make sure the user has sufficient privileges on the indices written to.

[BezVeze (МаКи БезВезе)]

Ive manage to connect successfully,

Ill use it this way

Thanks for the help man,