Logstash issues/ is not working

[Zay_Lin_Htun (R00TME)]

There are three AWS EC2 instances. First one is Elasticsearch+ Kibana, Second one is logstash and third one UbuntuOS which will send logs to logstash.

I deployed Elasticsearch and Kibana together on AWS EC2 instancens and I deployed Logstash EC2 instance and Linux EC2 instance which use filebeats to send log
All of the services are running properly but the main issue is that logstash, he can not receive the logs from filebeats so also can not send the logs to Elasticsearch.
This is testing environment and I allow all TCP port to avoid port filtering issues (not best practice for security)

there is no changes in Logstash.yml config file and **this is config for /etc/logstash/conf.d/beats.conf** 

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch { hosts => ["publicIP of Elasticsearch:9200"] }
  stdout { codec => rubydebug }
}

Logstash error

[INFO ] 2022-11-06 15:20:25.182 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[WARN ] 2022-11-06 15:20:25.814 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-11-06 15:20:26.832 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/beats.conf"}
[ERROR] 2022-11-06 15:20:26.846 [Agent thread] sourceloader - No configuration found in the configured sources.
[INFO ] 2022-11-06 15:20:27.230 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601, :ssl_enabled=>false}
[INFO ] 2022-11-06 15:20:27.315 [LogStash::Runner] runner - Logstash shut down.

Filebeat.yml Config

output.logstash:
  # The Logstash hosts
  hosts: ["public IP of logstash:5044"]

Elastic Search.yml config

network.host: 0.0.0.0
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["logstash Public IP"]

[leandrojmp (Leandro Pereira)]

How you installed Logstash and How are you running it?

You have this in your log:

[WARN ] 2022-11-06 15:20:25.814 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-11-06 15:20:26.832 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/beats.conf"}
[ERROR] 2022-11-06 15:20:26.846 [Agent thread] sourceloader - No configuration found in the configured sources.

It is looking for the configuration in /usr/share/logstash/beats.conf, not in /etc/logstash/conf.d/beats.conf.

Normally you would run logstash as a service and configure the pipelines in pipelines.yml.

[Zay_Lin_Htun (R00TME)]

This is how I install Logstash

>>wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

>>sudo apt-get install apt-transport-https

>>echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

>>sudo apt-get update && sudo apt-get install logstash

>>sudo su
>>nano /etc/logstash/conf.d/beats-input.conf

Beats.conf file Config

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch { hosts => ["publicIP of ElasticSearch:9200"] }
  stdout { codec => rubydebug }
}

And I run logstash with this command

bin/logstash -f /etc/logstash/conf.d/beats-input.conf --config.reload.automatic

[Zay_Lin_Htun (R00TME)]

@leandrojmp

/usr/share/logstash/beats.conf

Am I need to move this conf file to this file path?
/usr/share/logstash/beats.conf

[Zay_Lin_Htun (R00TME)]

This is my Current Config File Path

root@ip-10-10-10-15:/home/ubuntu# ls /etc/logstash/
conf.d  jvm.options  log4j2.properties  logstash-sample.conf  logstash.yml  pipelines.yml  startup.options
root@ip-10-10-10-15:/home/ubuntu# cat /etc/logstash/pipelines.yml 
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
root@ip-10-10-10-15:/home/ubuntu#