Stream log4j2 logs to logstash hosted on an azure VM via tcp

Elastic Stack Logstash
1

Hi All

I'm a bit new to the ELK stack so greatly appreciate any help

I have a mulesoft application on my local machine (looking to host the mule application on anypoint platform later) whose logs I am trying to send to logstash which I have running on an Azure ubuntu VM.

I first tried it on logstash hosted on my local using the TCP input for logstash and using the socket appender on mulesoft which worked fine.

Now when I'm trying to do the same thing but instead, now trying to connect to the VM instance instead of the local instance, it cannot connect to logstash

Here is a description of what my log4j2.xml socket looks like

<Socket name = "Socket" host = "IP address of the VM" port = "The configured port I have opened up in the VM firewall for this (same as the one used for local logstash)">
        	<JsonLayout compact = "true" eventEol = "true" />
        </Socket>

And the following is a description of what my logstash conf file looks like

input {
  tcp {
    port => the port I'm using for this
    host => "the ip address of the VM"
        codec =>json
  }
}

filter{
        date{
        match => ["timeMillis", "UNIX_MS"]
        }
        }

output {
  elasticsearch {
    the info for the elastic instance, this works fine, i tested using stdin to check
  }
}

As mentioned earlier, I'm new to logstash so please let me know if I'm missing something or doing something wrong.

Thank you

2

This seems to be a network issue and not a Logstash issue.

What is this IP? This is the internal IP of the VM? I would recommend that you remove the host option to use the default, which is 0.0.0.0, this means that this input will listen on all ip address available in the VM.

After that, test the telnet connection from your local machine to your VM, keep in mind that in this step you need to use the public ip address that your VM is using.

If this does not work, then you need to validate your network configuration.

3

Hi @leandrojmp

Thank you for replying

So the IP I have specified is the Private IP address of the VM, but with the way we have our network set up, the local machine is included in the VPC which would enable it to access the VM using the Private IP. So does logstash specifically need to use a Public IP to access the VM or am I correct in assuming that logstash should be able to connect to the VM using the Private IP?

As for this, earlier, I did not have the host specified leaving it to default to 0.0.0.0 in the config, I tried specifying it as 0.0.0.0 as well, but in both these scenarios too, I'm getting the same result.

4

No, you configure logstash to use 0.0.0.0, it will then listen on the private IP of your VM, your provide will then redirect the traffic from the public IP associated to your VM to the private IP.

You will then configure your local machine tha is running your application to send logs to the public ip.

Something like this:

Local Machine --> public ip on your cloud provider --> your cloud provider redirect the traffic --> private ip of your logstash vm.

If logstash is running and listening on the specified port on the private IP of the VM, and you cannot connect to it, then you need to validate your network configuration.

On logstash side you just need this in your input:

input {
    tcp {
        port => "port"
    }
}

The communication from your local machine to this logstash server depends on your network configuration and is not a logstash issue.