There are three AWS EC2 instances. First one is Elasticsearch+ Kibana, Second one is logstash and third one UbuntuOS which will send logs to logstash.
I deployed Elasticsearch and Kibana together on AWS EC2 instancens and I deployed Logstash EC2 instance and Linux EC2 instance which use filebeats to send log
All of the services are running properly but the main issue is that logstash, he can not receive the logs from filebeats so also can not send the logs to Elasticsearch.
This is testing environment and I allow all TCP port to avoid port filtering issues (not best practice for security)
there is no changes in Logstash.yml config file and **this is config for /etc/logstash/conf.d/beats.conf**
input {
beats {
port => 5044
}
}
output {
elasticsearch { hosts => ["publicIP of Elasticsearch:9200"] }
stdout { codec => rubydebug }
}
Logstash error
[INFO ] 2022-11-06 15:20:25.182 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[WARN ] 2022-11-06 15:20:25.814 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-11-06 15:20:26.832 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/beats.conf"}
[ERROR] 2022-11-06 15:20:26.846 [Agent thread] sourceloader - No configuration found in the configured sources.
[INFO ] 2022-11-06 15:20:27.230 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601, :ssl_enabled=>false}
[INFO ] 2022-11-06 15:20:27.315 [LogStash::Runner] runner - Logstash shut down.
Filebeat.yml Config
output.logstash:
# The Logstash hosts
hosts: ["public IP of logstash:5044"]
Elastic Search.yml config
network.host: 0.0.0.0
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["logstash Public IP"]