Need to bind logstash to eth0

[Nie]

I'm wondering why I am not seeing 5044 listener on the logstash server. I am seeing tcp port listeners on 9200 and 9600 when I execute "netstat -na". However, it's bound to loopback ip which is 127.0.0.1.

Also, I can't find where 9600 is defined. I've searched in /etc/logstash. I saw 9600 but is commented out.

Any help would be great appreciated.

Thanks!

[Badger]

9600 is the logstash Monitoring API. It is controlled in logstash.yml. For very recent versions it is api.http.host, for older versions I think it was just http.host.

9200 is the default port Elasticsearch listens on. network.host in Elasticsearch.yml determines the address it binds to.

[Nie]

Got it. Since we are building a logstash server and we'd like our partner/vendor to transmit data to this server, which port should logstash be listening on?

[Badger]

You should design the transport before you choose the port. lumberjack (beats)? http (either push or pull)? Will you have a buffer like kafka in front of logstash? There are all kinds of options.

[yaauie (Ry Biesemeyer)]

The http api is for monitoring Logstash itself and some minor control of logging features. It is bound to the loopback interface by default and is not meant to be publicly accessible to the network unless secured.

If you are looking to ingest data over http, you will need to stand up a pipeline with one or more HTTP input plugins, which you will be able to configure to bind to the interface and port of your choosing. Additionally, you will be able to configure it with a map of codecs which it will use to turn the inbound payloads into events for the pipeline.

[Nie]

The vendor will be pushing to our logstash.

Also the person who takes care of/administers logstash at work is gone for holidays. He provided us few .conf files which I placed(from his instructions) in /etc/logstash/conf.d directory and he gave me few commands that he said I should call during boot up of the linux machine.

Example:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/data.conf

input {
    http {
        host => "127.0.0.1"
        port => 9200
    }
}
filter {}
output {
    elasticsearch {
        hosts => ["https://w.x.y.z:9243"]
        index => "dynatrace"
        id => "dynatrace_logs"
        manage_template => false
        template_overwrite => false
        user => "elastic"
        password => "***************"
    }
}

I have no clue how it works. I do see a 127.0.0.1:9200 listener.

It looks weird to me that the command has to be executed. I was thinking that when logstash restarts, it should know these files. He didn't explain it well. He just sent me instructions via email.

Btw, I got the listener bound to all interface 9600.

So normally, which port would a client transmit to? Should it be 9200?

[Nie]

I think the .conf file that was given to me which I posted is for ingesting data. From what I can tell from this code, this matches whatever I see in the output of netstat -na specifically the listener.

input {
    http {
        host => "127.0.0.1"
        port => 9200
    }
}

and then below is where to forward the traffic to.

output {
    elasticsearch {
        hosts => ["https://w.x.y.z:9243"]
        index => "dynatrace"
        id => "dynatrace_logs"
        manage_template => false
        template_overwrite => false
        user => "elastic"
        password => "***************"
    }
}

I'll change host => "127.0.0.1" to host => "0.0.0.0" so that I can expose it to our load balancer.

Btw, I ran the command by passing that with -f. I got this error

command: /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/data.conf --path.settings /etc/logstash

[2021-12-17T01:45:50,436][FATAL][logstash.runner          ] Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

[Nie]

I found out that once a .conf file is in /etc/logstash/conf.d directory, it will pick it up. I now have non loopback 9200 http port running. So I am assuming all other conf files in this directory will be started. Am I right?

[yaauie (Ry Biesemeyer)]

Just as the log message states, there is another Logstash instance running on your machine, pointing to the same path.data. you can either stop the running process in the usual way, or specify --path.data pointing to a different readable/writable directory.

[Nie]

By default, where is --path.data located? I tried finding in the output of ps ax|grep logstash but it wasn't there in the long output.

[leandrojmp (Leandro Pereira)]

By default path.data is defined in the file logstash.yml, the location of this file will depend on how logstash was installed in the system.

If logstash was installed using package managers, like yum or apt, then logstash.yml will be located in /etc/logstash.

What where the commands that you got to run on boot?

This command for example will run Logstash and start the pipeline from the file data.conf.

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/data.conf

From your original question, you will need to change the host config to use 0.0.0.0, this will make logstash bind to every ip available in the server.

Just one tip, using 9200 as an input port in Logstash can lead to confusion as this port is also used by Elasticsearch.

[Nie]

Just one tip, using 9200 as an input port in Logstash can lead to confusion as this port is also used by Elasticsearch.

I knew it! I was wondering why they were using 9200. This 9200 is defined in data.conf specifically in the input section. This one right? Normally, http injestion, should I just use regular famous port like 8080?

input {
    http {
        host => "0.0.0.0"
        port => 9200
    }
}