I have installed the Elastic search, Kibana and Logstash to integrate with my application. I installed Kibana and Logstash as windows services by using winsw as mentioned below
Kibana :
Logstash:
With this I can start Kibana and Logstash and working as expected. But When I installed same in Remote machine in production environment, I observed that Kibana and Logstash are being stopped automatically at night time. Fo example when I start these applications it will run and after couple of hours like late night these are being stopped and I couldn't see any error logs for both of them. I initially though of issues with memory in remote machines and I increased the memory of the machine to 64gb ram, and I could see while running these services the memory usage is less than 30% of full memory. I tried with lot of options but I ended up with the same error.
As there are no error logs, unable to figure out the root cause.
Below are recent logs from both Kibana and Logstash and looks good and no error.
Kibana:
[2024-09-19T22:33:55.370+02:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"elasticsearch"}
[2024-09-19T22:48:55.636+02:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"elasticsearch"}
[2024-09-19T23:03:56.139+02:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"elasticsearch"}
[2024-09-19T23:18:56.625+02:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"elasticsearch"}
[2024-09-19T23:33:35.955+02:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found
[2024-09-19T23:33:47.947+02:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task
[2024-09-19T23:33:56.967+02:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"elasticsearch"}
[2024-09-19T23:48:57.431+02:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"elasticsearch"}
Logstash:
{
"logger_name" => "org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver",
"@timestamp" => 2024-09-05T12:39:12.316348Z,
"level" => "INFO",
"serviceName" => "download-service",
"thread_name" => "main",
"@Version" => "1",
"message" => "Exposing 1 endpoint beneath base path '/actuator'"
}
{
"logger_name" => "org.springframework.boot.web.embedded.tomcat.TomcatWebServer",
"@timestamp" => 2024-09-05T12:39:12.374350300Z,
"level" => "INFO",
"serviceName" => "download-service",
"thread_name" => "main",
"@Version" => "1",
"message" => "Tomcat started on port 8095 (http) with context path '/'"
}
{
"logger_name" => "com.zf.dwe.download.DownloadService",
"@timestamp" => 2024-09-05T12:39:12.395352400Z,
"level" => "INFO",
"serviceName" => "download-service",
"thread_name" => "main",
"@Version" => "1",
"message" => "Started DownloadService in 54.799 seconds (process running for 55.663)"
}
I'm not understanding what is the reason for this weird behavior. Could anyone please help on this.