Netflow Module Not Creating ES Index

[etsai7]

Hello,

When I launch logstash 6.5.4 with the netflow module, using the line:

bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055

it does not seem to create an ES index for Netflow as the docs suggest when I look in the Index Management on Kibana. The index is created for Kibana however.

My logstash.yml file:

modules:
  - name: netflow
    var.input.udp.port: 2055
    var.elasticsearch.hosts: "10.100.2.2:9200"
    var.kibana.host: "10.100.2.2:5601"
    var.elasticsearch.ssl.enabled: false
    var.kibana.scheme: http
    var.kibana.ssl.enabled: false
    var.kibana.ssl.verification_mode: disable

Is there an issue with my configuration?

Thanks,
Eric

[yaauie (Ry Biesemeyer)]

Is Logstash receiving a flow? In Elasticsearch, an index is created the first time a document is written, so if no data is flowing yet it would make sense for there to be no index.

[etsai7]

Hi Yaauie,

Thanks for getting back to me.

The Netflow Data should be flowing on our network, but may need to check if it is getting into the node that Logstash is hosted on. I wanted to confirm the following:

The server Logstash is on has 2 IPs, 10.100.1.1 and 10.2.1.1 for example. When I start logstash with above command, I receive this printout (partial printout):

[2019-05-09T11:04:50,925][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"module-netflow", :thread=>"#<Thread:0x305df264 run>"}
[2019-05-09T11:04:50,978][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2019-05-09T11:04:51,008][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:"module-netflow"], :non_running_pipelines=>[]}
[2019-05-09T11:04:51,056][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[2019-05-09T11:04:51,359][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I wanted to confirm that Logstash, with the Netflow module should be listening on those 2 IP addresses on Port 2055 correct, due to the 0.0.0.0 address property? If this set up is correct, then it must be the flow of data not making it through.

Thanks,
Eric

[yaauie (Ry Biesemeyer)]

In the context of servers, 0.0.0.0 can mean "all IPv4 addresses on the local machine". If a host has two IP addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host is configured to listen on 0.0.0.0, it will be reachable at both of those IP addresses.

-- 0.0.0.0 - Wikipedia

[etsai7]

Hi Yaauie,

Thanks for the info. Then it seems that perhaps it must be the data being blocked with a Firewall or just not making it through to the Node.

Eric

[Badger]

You can use "netstat -an | grep 2055" (or "netstat -an | findstr 2055" on Windows) to see what addresses it is listening on.

I once worked on an operating system for which the TCP stack would not bind to addresses in 192.168/16 and 10/8 when binding to 0.0.0.0. On such a system you would have explicitly bind to an address.

[etsai7]

In that case, do you know what file and variable I need to modify for the IP? I have tried it with the logstash.yml file using the variable http.host: "10.2.2.1", but the UDP listener always listens on 0.0.0.0

[Badger]

Apparently the module does not allow you to configure the host.

[etsai7]

I also realize that the module IP is not configurable, only the port it seems. However, it seems that this should be a logstash input variable logstash.inputs.udp. I am just not sure where I need to configure this.

[2019-05-10T13:05:36,861][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:4739"}
[2019-05-10T13:05:36,880][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:"module-netflow"], :non_running_pipelines=>[]}
[2019-05-10T13:05:36,941][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:4739", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[2019-05-10T13:05:37,246][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

If I wanted to bind the IP of Logstash itself (excluding module), is it not http.host that will allow it to listen on the specified IP?

Thanks,
Eric

[etsai7]

Hello,

Just an update, I was able to resolve this issue. Turns out it was not the binding issue but the Linux Firewall preventing packets from entering through a certain port. My original configuration was non-problematic.

Thanks for all the help!

Eric