Logstash upgrade error

Elastic Stack Logstash
1

Hello,

We've had some trouble when upgrading logstash from version 2.1 => 2.2. The system we're running on is Ubuntu 14.

Following the upgrade, the configuration failed. I had assumed there may be some syntax errors, so I tried to isolate which configuration file was the culprit to which I discovered the issue were with the output configuration.

output {
elasticsearch {
hosts => ["es01","es02","es03"]
workers => "30"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout { codec => rubydebug }
}

^ configuration used, and when directly checked for error:

root@log01:/opt/logstash/bin# ./logstash --configtest -f /etc/logstash/conf.d/99-output.conf
The error reported is:
comparison of String with 1 failed

I get the above error, and this is the error which appears in the logstash log, as well as the when trying to start the service.

Any assistance would be useful,

Some other steps I've taken include purging the install *(apt-get purge) and re-installed with new configuration, still the same occurs.

Kind regards,

David

2

Still working on this, trying to replicate issue on a openstack VM / see if it re-occurs on a different solution.

If you require further information regarding the problem posted let me know,

David

3 
/opt/logstash/bin/logstash --configtest -f test.conf 
es01: unknown error {:class=>"Manticore::ResolutionFailure", :level=>:error}
The error reported is: 
  comparison of String with 1 failed

It looks like the Manticore library is trying to resolve the host names in DNS while the config file is parsed. That doesn't seem right.

4

Thanks for taking a look Magnus, I've gone ahead and attempted to remove DNS resolution from the process and switched out my hostnames for IP addresses:

root@eslog01:/etc/logstash/conf.d# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/99-output.conf
The error reported is:
comparison of String with 1 failed

Still getting errors, I've added verbose and debug flags in an attempt to get more info:

root@eslog01:/etc/logstash/conf.d# /opt/logstash/bin/logstash --debug --verbose --configtest -f /etc/logstash/conf.d/99-output.conf
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"*", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"*", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"*", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"*", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"*", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"*", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"*", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"*", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx"], :level=>:info}
The error reported is:
  comparison of String with 1 failed
5

Hello,

Having investigated this further, the resolution of the issue was that it appears version 2.2 appears to be enforcing types of data more accurately:

So whilst the line :

workers => "30"

Had worked in 2.1, in 2.2 this had been causing the failure and was in fact meant to be declared as an integer:

workers => 30

Thanks for taking a look, this can be considered closed.

Regards,

David