Logstash only starts on 127.0.0.1

[fabricioraphael (fabrício raphael)]

I'am using Logstash 7.3 installed locally on server with:

sudo apt-get update && sudo apt-get install logstash

The goal is use filebeat to send logs to Elastic from Logstash but when I run Logstash, they starts on 127.0.0.1 and its not accessible by external machine..

When I run:
$ /usr/share/logstash/bin/logstash -e "" --log.level=debug
I see:
[DEBUG] 2019-10-15 18:23:14.272 [LogStash::Runner] runner - http.host: "127.0.0.1" [DEBUG] 2019-10-15 18:23:14.273 [LogStash::Runner] runner - http.port: 9600..9700 [DEBUG] 2019-10-15 18:23:14.273 [LogStash::Runner] runner - http.environment: "production"

And when I run netstat:
$ netstat -na |grep 9600
I get:
tcp 0 0 127.0.0.1:9600 0.0.0.0:* LISTEN

On my /etc/logstash/logstash.yml I have:

http.host: "10.198.9.233"
http.port: 9600-9700

I already tried to put any address on http.host like the real machine IP, 0.0.0.0, with no space indentation, 1 space, 2 spaces.. with tab but its not working.. I'am thinking witch Logstash is not applying my configs on /etc/logstash/logstash.yml

Anybody can help me? Thx

[Badger]

I think that is the monitoring interface, which means it is not directly relavent for ingestion.

Does your logstash configuration contain a beats input?

[TechGeekNZ (Andrew)]

From the documentation here

Use the given string as the configuration data. Same syntax as the config file. If no input is specified, then the following is used as the default input: input { stdin { type => stdin } } and if no output is specified, then the following is used as the default output: output { stdout { codec => rubydebug } } . If you wish to use both defaults, please use the empty string for the -e flag. The default is nil.

The way I read this is that as there is no external input or output, that it would make sense for logstash to start only using a loopback interface (I can replicate this behavior in my test environment)

Have you tried loading it by specifying the configuration file?

[fabricioraphael (fabrício raphael)]

@Badger and @TechGeekNZ

Yes, I have the pipeline configured on:
/etc/logstash/conf.d/

And yes, I already tried to run logstash by specifying the configuration file, its works fine but Logstash continues starts locally on "127.0.0.1"

Everything works fine, all pipeline files are working very well and sending data to Elastc. The only problem is that logstash is starting locally on 127.0.0.1 instead of 0.0.0.0 or 10.198.9.233.

Thx!

[TechGeekNZ (Andrew)]

I see, that puts things into better perspective

Is it possible to paste the full debugging logs. Have a look through the logs and see if you can find any errors around loading the logstash.yml file. Also, what are the permissions on the file?

[fabricioraphael (fabrício raphael)]

Oh, sorry for the delay,

No, I haven't see any errors analysing the logs. And the file permissions is ok..

$ /usr/share/logstash/bin/logstash --log.level=debug
Debug logs:
Thread.exclusive is deprecated, use Thread::Mutex [DEBUG] 2019-10-16 16:17:21.751 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [DEBUG] 2019-10-16 16:17:21.757 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x440dc4f7 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>} [DEBUG] 2019-10-16 16:17:21.759 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [DEBUG] 2019-10-16 16:17:21.759 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x5a342baf @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>} [DEBUG] 2019-10-16 16:17:22.205 [LogStash::Runner] runner - -------- Logstash Settings (* means modified) --------- [DEBUG] 2019-10-16 16:17:22.205 [LogStash::Runner] runner - node.name: "elasticsearch.intranet.explora" [DEBUG] 2019-10-16 16:17:22.205 [LogStash::Runner] runner - path.data: "/usr/share/logstash/data" [DEBUG] 2019-10-16 16:17:22.205 [LogStash::Runner] runner - modules.cli: [] [DEBUG] 2019-10-16 16:17:22.205 [LogStash::Runner] runner - modules: [] [DEBUG] 2019-10-16 16:17:22.205 [LogStash::Runner] runner - modules_list: [] [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - modules_variable_list: [] [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - modules_setup: false [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - config.test_and_exit: false [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - config.reload.automatic: false [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - config.reload.interval: 3000000000 [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - config.support_escapes: false [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - config.field_reference.parser: "STRICT" [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - metric.collect: true [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - pipeline.id: "main" [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - pipeline.system: false [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - pipeline.workers: 4 [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - pipeline.batch.size: 125 [DEBUG] 2019-10-16 16:17:22.206 [LogStash::Runner] runner - pipeline.batch.delay: 50 [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - pipeline.unsafe_shutdown: false [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - pipeline.java_execution: true [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - pipeline.reloadable: true [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - pipeline.plugin_classloaders: false [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - path.plugins: [] [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - config.debug: false [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - *log.level: "debug" (default: "info") [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - version: false [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - help: false [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - log.format: "plain" [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - http.host: "127.0.0.1" [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - http.port: 9600..9700 [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - http.environment: "production" [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - queue.type: "memory" [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - queue.drain: false [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - queue.page_capacity: 67108864 [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - queue.max_bytes: 1073741824 [DEBUG] 2019-10-16 16:17:22.207 [LogStash::Runner] runner - queue.max_events: 0 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - queue.checkpoint.acks: 1024 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - queue.checkpoint.writes: 1024 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - queue.checkpoint.interval: 1000 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - queue.checkpoint.retry: false [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - dead_letter_queue.enable: false [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - dead_letter_queue.max_bytes: 1073741824 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - slowlog.threshold.warn: -1 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - slowlog.threshold.info: -1 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - slowlog.threshold.debug: -1 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - slowlog.threshold.trace: -1 [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore" [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore" [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - path.queue: "/usr/share/logstash/data/queue" [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue" [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config" [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - path.logs: "/usr/share/logstash/logs" [DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - xpack.management.enabled: false [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.management.logstash.poll_interval: 5000000000 [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"] [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.management.elasticsearch.username: "logstash_system" [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.management.elasticsearch.hosts: ["https://localhost:9200"] [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification_mode: "certificate" [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.monitoring.enabled: false [DEBUG] 2019-10-16 16:17:22.209 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]

[Badger]

I don't see any indication you have configured a beats input.

[TechGeekNZ (Andrew)]

From the debug, the following line;

[DEBUG] 2019-10-16 16:17:22.208 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"

It's looking in that directory for the logstash.yml file. You mentioned earlier that this file is located in /etc/logstash/logstash.yml

This would indicate that the configuration is not being picked up. You can specify the location of logstash.yml with the command line option --path.settings

Give that a go with the debug and see if it then picks up the settings file. When it's picking up the correct configuration file, expect to see something like this in the debug log;

[2019-10-17T10:23:34,395][DEBUG][logstash.runner          ] *http.host: "172.16.40.72" (default: "127.0.0.1")