Logstash Issue with netflow modul (5.6)

Elastic Stack Logstash
1

Hello

I installed Logstash 5.6 and X-pack. Now i try to to configure it as it is described in the link:

I configured it in /etc/logstash/logstash.yml

modules:
        - name: netflow
    #var.elasticsearch.hosts: "http://IP:9200"
    #var.elasticsearch.username: "elastic"
    #var.elasticsearch.password: "changeme"
    var.kibana.host: "10.250.254.168:5601"
    var.kibana.username: "elastic"
    var.kibana.password: "changeme"
    var.input.udp.port: 9996

I got always this error message in /var/log/logstash/logstash-plain.log

[2017-09-14T16:13:33,580][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-09-14T16:13:33,603][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-09-14T16:13:33,693][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.6.0-java/modules/arcsight/configuration"}
[2017-09-14T16:13:33,750][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "var.kibana.host" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:32:in `get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:64:in `set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:83:in `merge'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:83:in `merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:135:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:239:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

I absolutely have no idea why it happens. The same happens with var.elasticsearch.hosts or var.elasticsearch.username

Can somebody help me !
i am despairing

2

YAML is indent sensitive. You have over-indented the -name: netflow line

3

Thank you very much. it works now.
But i have now another problem

Why doesnt logstash or kibana create Dashboards for netflow ?

Can you help me ?

Greets Lumi

4

Did you run it with --setup the first time? You must start it with --setup the first time you run in order for it to push the Dashboards.

5

I think i have another problem. i analysed the Index of netflow..

It is the same as the index of syslog.

I don't have any netflow input, just Syslog input.

logstash.yml:

modules:
   - name: netflow
     var.elasticsearch.hosts: "http://xxx:9200"
     var.kibana.host: "xxx:5601"
     var.input.udp.port: "2056"

/etc/logstash/conf.d/pipeline.conf:

input {
    syslog {
           port => 1514
                  }
}

output {
         elasticsearch {
               hosts => [ "XXX:9200" ]
               index => "logstash-syslog-%{+YYYY.MM.dd}"

          }
}

Why does it create a new index "netflow-*" ?

image
image.png1501×191 13.6 KB

Can you help me ?

Greets
Lumi

6

In Logstash 5.x, you cannot run multiple pipelines at once. Since you defined the modules block in your logstash.yml, it will ignore any other pipeline you have defined, and only run the module you defined. In 5.x, you also cannot run more than one module at once.

The module has its own pipeline configuration file, ES template, and all that. You can take a look at it in the modules subdirectory of /usr/share/logstash.

When Logstash 6.0 is released, you'll be able to run multiple pipelines and modules concurrently in the same Logstash instance.

7

So where is the correct place for the netflow configuration - in logstash.yml, or a .conf file?

Looking through the /usr/share/logstash/modules/netflow/configuration/logstash directory, I can see a file netflow.conf.erb (on CentOS 7, using logstash-5.6.1-1.noarch.RPM). The file is very long and contains many <% ... %> blocks. Am I meant to configure this file by hand, then place it in /etc/logstash/conf.d? If not, how should I use it?

The README.md in /usr/share/logstash/modules/netflow mentions that one configuration variable should be placed in logstash.yml (var.input.udp.port), does this override the settings in the .conf file? Do I still need the .conf file?

I'm still not clear on the relationship between these two places for configuration. It seems like an odd design choice to allow some (but not all) configuration elements to be placed in a YML file, and then have those silently render other configuration elements to be ignored.

Any guidance appreciated.

8

You don't configure anything but either the command-line or the modules section of the logstash.yml file. The values you place there will determine what goes in the <%...%> blocks. The module is only meant to be configured in this manner. I directed you there so you could see what it contains, not change it.

As previously stated, this is an unfortunate side effect of 5.x only allowing a single pipeline to run at once. When Logstash 6.0 is released, you'll be able to run multiple pipelines, and it will read those other configurations as you have them. We wanted to release the modules feature in 5.6 to showcase it, but that required us limiting Logstash to one pipeline, so the behavior I outlined was the executive decision.

9

I alwys get this message:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2017-09-25 16:39:23.900 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::ConfigLoadingError: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["10.250.254.168:9200"] and Kibana hosts: ["10.250.254.168:5601"]>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:97:in pipeline_configs'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:272:in execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:in run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in (root)'"]}

Can you help me ?

10

This suggests a potential authentication error, or an incorrect host. Logstash is trying to connect to Elasticsearch to upload the index template, and to Kibana to upload the dashboards and visualizations. I don't see any more in depth information there, so it's hard to identify which failed. You could check the logs for either of those.

Also, the error has your hosts configured on the same IP, different ports. Is that accurate?

11

Good news, my netflow config started working after updating logstash to 5.6.2, using the RPMs provided at https://artifacts.elastic.co/packages/5.x/yum

:+1:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.