Connecting Render web service to Elasticsearch via TLS-enabled syslog drain

Elastic Observability
1

Hello,
I'm new to configuring Elasticsearch observability but my goal right now is to get syslogs from my web service setup on Render (render.com) into my Elasticsearch instance (Render log stream docs here). Is this even possible? I'm getting mixed messages from the elastic docs and am unclear on where to point the syslog drain if that can be used. It seems like Filebeats does something very similar to a log drain (or maybe it is a log drain?) which gives me hope that this is possible, but then I assume I'd need some module to connect to the drain? Apologies in advance for any lack of clarity, I'm still wrapping my head around this whole system. Happy to clarify anything!

2

Can you provide more context about your infrastructure? It is not clear.

You have a web service running on this Render provider and want to use the Render Log stream feature to send the logs of this service to Elasticsearch? Is that right?

If so, I'm not sure you can do that using Filebeat, but you can do that using Logstash, I have a similar setup, but it is not so simple.

You will need a public endpoint listening on TCP with a valid SSL certificate, how you will do that depends entirely on your infrastructure.

3

Thanks for the quick response! Render is similar to Heroku in that it's a Platform as a Service tool so I don't have much control over the infrastructure. The web server is running Node and I have shell access. What infra context are you looking for? I have no idea if I can get a public TCP endpoint setup but I can ask Render. Do you have any docs you could point me to?

4

Where is your Elasticsearch cluster running? You will need a place to run a Logstash instance and configure this istance to listen on TCP and receive this logs to send to Elasticsearch.

The Render Log Stream you shared send logs through the internet, so you need a public endpoint.

5

Ah sorry I'm running Elasticsearch on Elastic Cloud.

6

Yeah, Elastic Cloud does not offer the possibility to run Logstash, so you will need to run it elsewhere, not sure if you can run it on a VM on Render as I do not know this service, but you need someplace to run it.

The first thing is to define where you will run it and check the requirements, you will need to be able to open a public TCP port for Logstash to listen, on Logstash you will use the TCP input to open the port and use a SSL certificate, this certificate needs to be generated by a CA that render will trust, I think you can use Let's Encrypt to create the certificate.

Then you will receive the messages, parse the data and then send to Elastic Cloud.

7

So I guess what that means is I can't use the log drain with Logstash at all? Because my understanding of the log drain is that it's constantly pushing out logs over a TLS connection. Is there no way to get Logstash (or anything else that can transmit logs to Elasticsearch) to receive logs instead of having to listen on a port?

8

You can use Logstash, there are some confusion here.

The log stream on render will push logs over a TLS connection, this needs a destination IP Address and a Port, which will be in Logstash.

Look at the documentation on Render, you need to specify an endpoint in the format HOST:PORT.

click “Add Log Stream”, and enter your endpoint URL in the format HOST:PORT

So you need Logstash to listen on a public port to receive the data.

10

Thanks a lot. This is starting to make sense and I think I'm making some progress now!

I've made some progress getting things running locally and also on Render but one error I'm seeing on startup in both places is about the license reader. Have you ever seen this error before?

Feb 23 11:11:01 AM  [2023-02-23T16:11:01,423][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
Feb 23 11:11:01 AM  [2023-02-23T16:11:01,432][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}
Feb 23 11:11:01 AM  [2023-02-23T16:11:01,433][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"

The Logstash service starts up fine after reporting that, but I'm just trying to figure out what it means and if/how I can fix it.