Dockerized Logstash and netflow module install issue

iukea · August 3, 2019, 8:37pm

Hello,

I am currently having issues with installing the Netflow module inside of dockerized logstash

Docker file

FROM docker.elastic.co/logstash/logstash:7.3.0

ADD modules.yml /usr/share/logstash/config/
RUN bin/logstash --modules netflow --setup

Inside the modules.yml file

modules:
- name: netflow
  var.elasticsearch.hosts: "127.0.0.1:9200"
  var.elasticsearch.username: "elastic"
  var.elasticsearch.password: "changeme"
  var.kibana.host: "127.0.0.1:5601"
  var.kibana.username: "elastic"
  var.kibana.password: "changeme"

The log messages I am getting in the docker container

Step 12/12 : RUN bin/logstash --modules netflow --setup
 ---> Running in 313d7506c713
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-08-03T18:53:00,092][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2019-08-03T18:53:00,110][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2019-08-03T18:53:00,465][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-08-03T18:53:00,471][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.0"}
[2019-08-03T18:53:00,498][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"3cfe1c3c-ef79-458c-89dc-f428492b84c0", :path=>"/usr/share/logstash/data/uuid"}
[2019-08-03T18:53:00,990][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2019-08-03T18:53:01,826][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2019-08-03T18:53:02,017][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
[2019-08-03T18:53:02,135][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-08-03T18:53:02,144][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2019-08-03T18:53:02,187][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2019-08-03T18:53:02,334][INFO ][logstash.config.modulescommon] Setting up the netflow module
[2019-08-03T18:53:02,564][ERROR][logstash.modules.kibanaclient] Error when executing Kibana client request {:error=>#<Manticore::SocketException: Connection refused (Connection refused)>}
[2019-08-03T18:53:02,643][ERROR][logstash.modules.kibanaclient] Error when executing Kibana client request {:error=>#<Manticore::SocketException: Connection refused (Connection refused)>}
[2019-08-03T18:53:02,695][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in `block in pipeline_configs'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2572:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2019-08-03T18:53:02,703][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[2019-08-03T18:53:02,924][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-08-03T18:53:07,994][INFO ][logstash.runner          ] Logstash shut down.
The command '/bin/sh -c bin/logstash --modules netflow --setup' returned a non-zero code: 1

Any words of wisdom? Am I missing something dumb?

Christian_Dahlqvist · August 3, 2019, 9:46pm

Are you by any chance running an Elasticsearch cluster based on the OSS distribution rather than the default one?

iukea · August 3, 2019, 10:47pm

I don't think so. I am getting all my docker images from the default ElasticSearch Docker image hub.

for Logstash docker image I used this link
https://www.elastic.co/guide/en/logstash/current/docker.html

and to learn how to configure logstash I used
https://www.elastic.co/guide/en/logstash/current/docker-config.html

I also setup the passwords on the ElasticSearch server by going and running
bin/elasticsearch-setup-passwords interactive (in the elasticsearch server)

iukea · August 4, 2019, 5:50pm

@Christian_Dahlqvist
To anyone that finds this post I think I found the fix. Kinda over thought this whole process.

Set you logstash yml like so.

modules:
- name: netflow
  var.elasticsearch.hosts: "127.0.0.1:9200"
  var.elasticsearch.username: "elastic"
  var.elasticsearch.password: "changeme"
  var.kibana.host: "127.0.0.1:5601"
  var.kibana.username: "elastic"
  var.kibana.password: "changeme"

in your docker image file I removed this line

RUN bin/logstash --modules netflow --setup

once done check your docker logs of your logstash server and might get lucky and see.

Only took me like 10 hours to figure this mess out lol

sgreszcz · August 5, 2019, 9:42pm

I'm going to try this tomorrow, thanks!

Also, there is a strange thing where I've found that maybe logstash is overwriting the netflow router host field with (I think) a Docker IP address: "host": "172.19.0.1".

iukea · August 6, 2019, 4:32am

Check your docker-compose file that might be the issue there if you are exposing porting

sgreszcz · August 6, 2019, 3:01pm

Was a problem with logstash being on docker bridge network by default. I had to move it to host (to not overwrite the netflow source IP).

services:
logstash:
network_mode: host
image: "docker.elastic.co/logstash/logstash:{
hostname: "{{ansible_hostname}}"
container_name: logstash-netflow

There is a big thread on this:

github.com/docker/for-linux

preserving source IP when using bridge network

opened 08:57AM - 13 Dec 17 UTC

closed 10:53AM - 13 Dec 17 UTC

jensenja

* [x] This is a bug report * [ ] This is a feature request * [x] I searched existing issues before opening this one (I actually don't know if this is a bug/intended behavior/an error on my part. I also don't know if this is a Docker problem, or a Logstash problem, so I've cross-posted details here: https://discuss.elastic.co/t/logstash-in-docker-syslog-preserving-source-ips/111492)  ### Expected behavior I'm using `docker.elastic.co/logstash/logstash-oss:6.0.0` on a server that's mainly going to be used for receiving syslog and snmptrap data from primarily networking devices and some other servers. Mapping a privileged port (514) on my container host to a non-privileged port (5014) on the container should preserve source IP's of my devices that are shipping log lines to the container host. ### Actual behavior 1. Set up syslog/tcp/udp input with port 5014, then set my devices to send to the container host on port 5014. (`-p 5014:5014 -p 5014:5014/udp`)This appears to yield accurate results. 2. Set syslog/tcp/udp input with port 5014 and then run the container with `-p 514:5014 -p 514:5014/udp`. This yields some host IP's being set to the Docker bridge IP, but not all. 3. Set syslog/tcp/udp input with port 514 and then run the container with `-p 514:514 -p 514:514/udp --user=root`. This ALSO yields some IP's being hidden by the Docker bridge IP, but not all. 4. Set syslog/tcp/udp input with port 514 and just run the container with `--user=root --net=host`. This appears to have the desired effect but has the downside of opening said ports on ALL interfaces of my container host. (Yes I realize I can use `iptables` to mitigate this) *NB:* It doesn't matter if the `syslog` logstash input plugin is used, or just the `udp` input plugin. ### Steps to reproduce the behavior  See actual behavior. I'll attempt to translate to docker one-liners but it might be tough if you don't have a handful of devices/systems outside of the container host to send log messages from. I do have a reliable offending device though. My docker container host is on 10.15.2.11 and this particular switch has an IP of 10.3.3.31 For this use scenario I changed my device config to send syslog to 10.15.2.11 on UDP/5014 ``` mkdir pipeline echo 'input { udp { port => 5014 host => "0.0.0.0" } } filter { syslog_pri { } } output { stdout { codec => rubydebug } }' > pipeline/logstash.conf docker run -it --rm -p 5014:5014/udp -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash-oss:6.0.0 { "@timestamp" => 2017-12-13T08:37:48.241Z, "syslog_severity_code" => 5, "syslog_facility" => "user-level", "@version" => "1", "host" => "10.3.3.31", <---- correct IP "syslog_facility_code" => 1, "message" => "SYSLOG_MESSAGE_REDACTED", "syslog_severity" => "notice" } ``` Number 2. The `logstash.conf` file created in step 1 does not change. I change my device back to using the log host on the standard UDP/514. ``` docker run -it --rm -p 514:5014/udp -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash-oss:6.0.0 { "@timestamp" => 2017-12-13T08:45:56.166Z, "syslog_severity_code" => 5, "syslog_facility" => "user-level", "@version" => "1", "host" => "172.17.0.1", <---- WRONG IP "syslog_facility_code" => 1, "message" => "SYSLOG_MESSAGE_REDACTED", "syslog_severity" => "notice" } ``` Number 3. ``` echo 'input { udp { port => 514 host => "0.0.0.0" } } filter { syslog_pri { } } output { stdout { codec => rubydebug } }' > pipeline/logstash.conf docker run -it --rm -p 514:514/udp -v ~/pipeline/:/usr/share/logstash/pipeline/ --user=root docker.elastic.co/logstash/logstash-oss:6.0.0 { "@timestamp" => 2017-12-13T08:50:11.071Z, "syslog_severity_code" => 5, "syslog_facility" => "user-level", "@version" => "1", "host" => "172.17.0.1", <---- WRONG IP "syslog_facility_code" => 1, "message" => "SYSLOG_MESSAGE_REDACTED", "syslog_severity" => "notice" } ``` And last but not least: ``` docker run -it --rm --net=host -v ~/pipeline/:/usr/share/logstash/pipeline/ --user=root docker.elastic.co/logstash/logstash-oss:6.0.0 { "@timestamp" => 2017-12-13T08:52:11.321Z, "syslog_severity_code" => 5, "syslog_facility" => "user-level", "@version" => "1", "host" => "10.3.3.31", <---- correct IP "syslog_facility_code" => 1, "message" => "SYSLOG_MESSAGE_REDACTED", "syslog_severity" => "notice" } ``` **Output of `docker version`:** ``` Client: Version: 17.09.1-ce API version: 1.32 Go version: go1.8.3 Git commit: 19e2cf6 Built: Thu Dec 7 22:24:16 2017 OS/Arch: linux/amd64 Server: Version: 17.09.1-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: 19e2cf6 Built: Thu Dec 7 22:22:56 2017 OS/Arch: linux/amd64 Experimental: false ``` **Output of `docker info`:** ``` Containers: 2 Running: 2 Paused: 0 Stopped: 0 Images: 37 Server Version: 17.09.1-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0 runc version: 3f2f8b84a77f73d38244dd690525642a72156c64 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 4.9.0-4-amd64 Operating System: Debian GNU/Linux 9 (stretch) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 31.38GiB Name: va1-netops-bastion-01 ID: 7LKX:2PO5:N2ME:VLHH:Z6KQ:WBTD:GCRE:MN7M:YIM2:UPHC:3M4K:ABFR Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` **Additional environment details (AWS, VirtualBox, physical, etc.)** I'm running Debian 9.3 amd64 on a baremetal server. I have no special networking setup, aside from a couple extra IP'ed interfaces on the server, one of which is Internet-facing, which is why using `--net=host` is undesirable. `tcpdump` on the container host's interface confirms that the messages are coming in with the proper source IP on the wire.

github.com/moby/libnetwork

Docker changes source IP address from incoming external UDP packets

opened 06:52AM - 20 Oct 17 UTC

closed 03:15AM - 14 Dec 17 UTC

gabrielmocan

Dear colleagues, I'm currently dockerizing a traffic analyzer application whi…ch I developed in Python and it's already being used in production. I'm almost there, the first step I took was to create the image of my app based on Alpine 3.6. The main dependency for my code to run is the nfdump project (available at https://github.com/phaag/nfdump). This project delivers a collector for Netflow/IPFIX UDP packets called "nfcapd", which runs on port 9995 (properly exposed on my Dockerfile). The thing is, If I run the container with default network and publishing port 9995, the UDP packets coming from different routers (with different IPs, obviously) all come with the same docker gateway IP (172.18.0.1) -- but I need original source IP address in order to determine from which router the packets belongs to. So I searched for solutions and stumbled with the --net=host option. If I run the container with run command and passing --net=host, the source IP of the incoming UDP packets is not altered, solving my problem aparently. But then again, the thing is, this container is not running alone in the service. Inside the docker-compose.yml I've created some other services, such as db (InfluxDB) which has to be reachable by the collector container (it collects the packets, process them, and send the results to InfluxDB). Wrapping it up, If I use the host option for the collector container network, it receives the correct source IP addresses, but in the other hand, it cant communicate to the other containers which runs in the stack. The DNS does not work and I cant resolve the db container name from within the collector container. By obvious reasons I cant rely on IPs, as taught by you, professor, as I expect to run multiple instances of this service to serve different clients from a single machine (swarm is not a requirement in this project).

sgreszcz · August 6, 2019, 3:09pm

Were you able to get the default netflow dashboards working? I still have not, although my netflow/logstash pipleline is working well.

This command cannot be run from within the container:
/bin/sh -c bin/logstash --modules netflow --setup

as I get:
Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

iukea · August 6, 2019, 3:30pm

Not yet. Trying to figure that one out next.

Keep me updated on your progress also

Getting the same error as you right now

iukea · August 6, 2019, 9:18pm

after when you got it working did your of you .conf files stop working? etc/logstash/conf.d

right now I got Netflow Working but I am trying to get other conf files to be loaded into logstash.

Any idea?

iukea · August 6, 2019, 9:48pm

just imported the visualizations by hand .https://github.com/elastic/logstash/tree/master/modules/netflow/configuration/kibana/7.x
worked fine .

sgreszcz · August 7, 2019, 2:54pm

Wow, you imported all those JSON files in manually?

system · September 4, 2019, 2:54pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.