In need of some help, have newly come around the realm of AWS and the Elastic Cloud set up(s).
Have recently set up and configured an Logstash agent in an ec2 instance. On the output of this, I would like this to be able to output to the Elastic Cloud set up so the data is able to be stored in the Eleasticsearch database and therefore seen in the Kibana console.
I can curl to the elastic cloud instances from the ec2 instance within AWS but cannot output using Logstash on the ec2 instances.
For the Eleasticsearch in the Elastic Cloud, this does come with X-Pack on deployment as the security features, centralized pipeline management etc is seen under the management area within Kibana.
For the config of the Logstash agent which is built on the ec2 instance. Have tried both using the hostname, user and password configs and have also attempted with the config of using the cloud.id and cloud.auth.
As I can curl from the cli on the logstash agent to the elastic cloud and get the elasticsearch response from it. Would this probably no be a port allocation issue? or would it when going via logstash as when looking have seen this comment being made
if it is using
cloud.idfor monitoring then that's the problem because it gives the wrong URL in ECE (it's designed for ESS) as you can see by base64 decoding it .. it uses
:443as the port not
:9243(which is the ECE default). So To make applications play nice with
cloud.idin ECE you have to use (eg)
Within the Logstash.yml from the xpack setting the certs needed will need to be added when/ if the ssl=true?
When the error of
LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach live Elasticsearch cluster
This means that there needs to be a manual way to set X-Pack correctly on Elastic Cloud? or would this be Logstash? If on Logstash, with the certs from elasticsearch from the elastic cloud how would you be able to do this/ get this?
If in turn, the would need be, to create a logstash instances within the elastic cloud environment. Then on the ec2 instance output logstash to the elastic cloud logstash which will then internally push it to elasticsearch?