Elastic 7.0 and Netflow module : no more indices created since upgrade to 7.0

Elastic Stack Logstash
1

Hi,

I had elastic stack 5.6.4 running and netflow module used with logstash. Every thing was working normally.

I just upgraded to 7.0 and saw that the Netflow module couldn't work any more : No more new netflow* indices are created.

Here's what I have in the logstash logs.
I have a lot of warning of the same type ("Could not index event to Elasticsearch") and sometimes an ERROR ("An unknown error occurred sending a bulk request to Elasticsearch") :

(See logs and configuration in the following messages of this thread)

2

Here are the logs of Logstash :

//////////////////////////////////// LOGS of Logstash ///////////////////

[2019-05-09T19:32:10,759][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.0.1"}
[2019-05-09T19:32:13,446][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2019-05-09T19:32:13,448][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2019-05-09T19:32:13,532][INFO ][logstash.config.modulescommon] Starting the netflow module
[2019-05-09T19:32:28,692][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-05-09T19:32:28,715][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-05-09T19:32:28,721][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
   
  
[2019-05-09T19:32:30,806][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"module-netflow", "pipeline.workers"=>5, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>625, :thread=>"#<Thread:0x211c23cd run>"}
[2019-05-09T19:32:30,871][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"module-netflow"}
[2019-05-09T19:32:31,178][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2019-05-09T19:32:31,246][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:"module-netflow"], :non_running_pipelines=>[]}
[2019-05-09T19:32:31,383][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[2019-05-09T19:32:31,415][INFO ][logstash.config.modulescommon] Starting the netflow module

//////   A Lot of warnings of this type : 
[2019-05-09T19:33:05,342][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netflow-2019.05.09", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x3f26467>], :response=>{"index"=>{"_index"=>"netflow-2019.05.09", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [netflow-2019.05.09]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}

////// And sometimes an error of this type : 

[2019-05-09T19:34:58,580][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"bignum too big to convert into `long'", :error_class=>"LogStash::Json::GeneratorError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:27:in `jruby_dump'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:119:in `block in bulk'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:119:in `block in bulk'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:117:in `bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:286:in `safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:191:in `submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:159:in `retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:235:in `block in start_workers'"]}


////////   END Excerpt  of logs
3

Here's my logstash config file :

(Only the uncommented lines)

////////////////// Config file ///////

node.name: xxx.xxxx.xxxx
path.data: /usr/share/logstash/data_modules/
pipeline.id: modules
pipeline.workers: 5
modules:
  - name: netflow
path.logs: /var/log/logstash_modules/
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
xpack.monitoring.collection.pipeline.details.enabled: true

/////////

4

I have an old template for this kind of indices (netflow*) called "netflow_template" and apparently version 7.0 created another one "netflow" :

These two templates are used in elasticsearch.

///////// Old netflow template /////////
{
  "netflow_template" : {
    "order" : 10,
    "version" : 60003,
    "index_patterns" : [
      "netflow-*"
    ],
    "settings" : {
  "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}
//////  End first template

Second template :

{
  "netflow" : {
    "order" : 0,
    "index_patterns" : [
      "netflow-*"
    ],
    "settings" : { },
    "mappings" : { },
    "aliases" : { }
  }
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

7

@samia: Can you check to see if you have any index templates in Elasticsearch for netflow? You can do this by hitting the _cat API:

GET /_cat/templates

If you see any related to netflow, you may want to see if they include any default mappings. You can use the template API to get these:

GET _template/netflow

If there are any default mappings, you'll want to remove those by updating the template.