I am testing elastic search with logstash and filebeats, trying to import our IIS log for analysing and viewing in Kibana. All latest version (5.0).
I have run into a problem during the startup of logstash where it says this during startup:
[2016-11-02T09:57:12,578][WARN ][logstash.runner ] SIGINT received. Shutting down the agent.
[2016-11-02T09:57:12,687][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2016-11-02T09:58:44,305][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"192.168.0.2:5045"}
[2016-11-02T09:58:44,633][INFO ][org.logstash.beats.Server] Starting server on port: 5045
[2016-11-02T09:58:45,008][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>["http://192.168.0.2:9200"]}}
[2016-11-02T09:58:45,027][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"M:\logstash-5.0.0\config\iis.myapp.template.json"}
[2016-11-02T09:58:45,258][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>{"enabled"=>false}}, "dynamic_templates"=>[{"iisTemplate"=>{"mapping"=>{"doc_values"=>true, "ignore_above"=>1024, "index"=>"not_analyzed", "type"=>"{dynamic_type}"}, "match"=>""}}], "properties"=>{"@timestamp"=>{"type"=>"date", "index"=>"analyzed"}, "status"=>{"type"=>"integer", "index"=>"analyzed", "doc_values"=>true}, "timeTaken"=>{"type"=>"integer", "index"=>"analyzed", "doc_values"=>true}, "bytesSent"=>{"type"=>"integer", "index"=>"analyzed", "doc_values"=>true}, "bytesTotal"=>{"type"=>"integer", "index"=>"analyzed", "doc_values"=>false}, "bytesReceived"=>{"type"=>"integer", "index"=>"analyzed", "doc_values"=>true}, "subStatus"=>{"type"=>"integer", "index"=>"not_analyzed"}, "geoip"=>{"type"=>"object", "dynamic"=>"true", "index"=>"analyzed", "properties"=>{"location"=>{"type"=>"geo_point"}}}}}, "settings"=>{"index.refresh_interval"=>"5s"}, "template"=>"iis-"}}}
[2016-11-02T09:58:45,274][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2016-11-02T09:58:45,305][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contact Elasticsearch at URL 'http://192.168.0.2:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError"}
[2016-11-02T09:58:45,321][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["192.168.0.2:9200"]}
[2016-11-02T09:58:45,586][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"M:\logstash-5.0.0\GeoLite2-City.mmdb"}
[2016-11-02T09:58:45,618][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-11-02T09:58:45,618][INFO ][logstash.pipeline ] Pipeline main started
[2016-11-02T09:58:45,774][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Can anyone tell me from this what the issue could be? Elastic pointed me to this forum...
The template is present in the log above.
Also, the default configuration of elasticsearch itself seems to cause it to crash (multiple exceptions and out-of-memory errors). I had hoped elasticsearch could handle it a bit better when memory was tight instead of outright failing. :-/
Thanks,
Bernhard