I'm trying to get our Geo location filter to work again after patching from v.2.4 to v.5.6.
First I found that I needed manually to convert our geopoint data to floats where previous they were float like their sources (%long + %lat) in add_field.
But can't get the ASN number and organization pull from ASN mmdb and wonder why.
All hints appreciated, TIA!
[2017-11-07T16:13:12,659][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/GeoIP2/GeoLite2-City.mmdb"}
[2017-11-07T16:13:12,680][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/GeoIP2/GeoLite2-ASN.mmdb"}
[2017-11-07T16:13:12,685][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x4edd1428 @metric_events_out=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"930b071d4d201c7f5535b660f85c28c9e4877f6c-40\", :events] key: out value: 0, @metric_events_in=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"930b071d4d201c7f5535b660f85c28c9e4877f6c-40\", :events] key: in value: 0, @logger=#<LogStash::Logging::Logger:0x55b30f83 @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x3eb81452>>, @metric_events_time=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"930b071d4d201c7f5535b660f85c28c9e4877f6c-40\", :events] key: duration_in_millis value: 0, @id=\"930b071d4d201c7f5535b660f85c28c9e4877f6c-40\", @klass=LogStash::Filters::GeoIP, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x1e081669 @metric=#<LogStash::Instrument::Metric:0x4d70d633 @collector=#<LogStash::Instrument::Collector:0xf21709 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x29415c59 @store=#<Concurrent::Map:0x00000000065a5c entries=2 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x7724bc46>, @fast_lookup=#<Concurrent::Map:0x00000000065a60 entries=228 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"930b071d4d201c7f5535b660f85c28c9e4877f6c-40\", :events]>, @filter=<LogStash::Filters::GeoIP default_database_type=>\"ASN\", database=>\"/usr/share/GeoIP2/GeoLite2-ASN.mmdb\", source=>\"[@metadata][geoip]\", target=>\"geoasn\", fields=>[\"autonomous_system_number, autonomous_system_organization\"], add_field=>{\"[geoasn][asn]\"=>\"%{[geoasn][autonomous_system_organization]}\", \"[geoasn][number]\"=>\"%{[geoasn][autonomous_system_number]}\"}, id=>\"930b071d4d201c7f5535b660f85c28c9e4877f6c-40\", enable_metric=>true, periodic_flush=>false, cache_size=>1000, lru_cache_size=>1000, tag_on_failure=>[\"_geoip_lookup_failure\"]>>", :error=>"illegal field value autonomous_system_number, autonomous_system_organization. valid values are [AUTONOMOUS_SYSTEM_NUMBER, AUTONOMOUS_SYSTEM_ORGANIZATION, CITY_NAME, COUNTRY_NAME, CONTINENT_CODE, CONTINENT_NAME, COUNTRY_CODE2, COUNTRY_CODE3, IP, ISP, POSTAL_CODE, DMA_CODE, REGION_NAME, REGION_CODE, TIMEZONE, LOCATION, LATITUDE, LONGITUDE, ORGANIZATION]"}
[2017-11-07T16:13:13,153][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>java.lang.IllegalArgumentException: illegal field value autonomous_system_number, autonomous_system_organization. valid values are [AUTONOMOUS_SYSTEM_NUMBER, AUTONOMOUS_SYSTEM_ORGANIZATION, CITY_NAME, COUNTRY_NAME, CONTINENT_CODE, CONTINENT_NAME, COUNTRY_CODE2, COUNTRY_CODE3, IP, ISP, POSTAL_CODE, DMA_CODE, REGION_NAME, REGION_CODE, TIMEZONE, LOCATION, LATITUDE, LONGITUDE, ORGANIZATION], :backtrace=>["org.logstash.filters.Fields.parseField(org/logstash/filters/Fields.java:78)", "org.logstash.filters.GeoIPFilter.createDesiredFields(org/logstash/filters/GeoIPFilter.java:95)", "org.logstash.filters.GeoIPFilter.<init>(org/logstash/filters/GeoIPFilter.java:71)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/lib/logstash/filters/geoip.rb:119)", "RUBY.register(/usr/share/logstash/vendor/jruby/lib/ruby/1.9/forwardable.rb:201)", "RUBY.register_plugin(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235)", "RUBY.start_pipeline(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:398)"]}
[2017-11-07T16:13:13,178][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>19600}
[2017-11-07T16:13:16,164][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
These are the GeoLite2 mmdb files:
# ls -l /usr/share/GeoIP2/
total 55948
-rw-r--r-- 1 root root 55 Oct 4 23:53 COPYRIGHT.txt
-rw-r--r-- 1 root root 5811489 Oct 30 01:12 GeoLite2-ASN.mmdb
-rw-r--r-- 1 root root 51469823 Oct 4 23:53 GeoLite2-City.mmdb
-rw-r--r-- 1 root root 433 Oct 4 23:53 LICENSE.txt
This is our Geo location filter:
#####################################################
#
# GeoIP lookups
#
#####################################################
filter {
# Are geoip defined? => try to lookup its GeoIP values
if '[@metadata][geoip]' {
geoip {
default_database_type => 'City'
database => '/usr/share/GeoIP2/GeoLite2-City.mmdb'
cache_size => 5000
source => '[@metadata][geoip]'
target => 'geoip'
fields => ['city_name','continent_code','country_name','latitude','longitude']
add_field => [ '[geoip][location]', '%{[geoip][longitude]}' ]
add_field => [ '[geoip][location]', '%{[geoip][latitude]}' ]
remove_field => ['[geoip][longitude]', '[geoip][latitude]']
add_field => [ '[geoip][ip]', '%{[@metadata][geoip]}' ]
}
# coordinates are not float anymore in v.5.6 as it used to be in v.2.4 but strings, hence
mutate {
convert => { '[geoip][location]' => 'float' }
}
geoip {
default_database_type => 'ASN'
database => '/usr/share/GeoIP2/GeoLite2-ASN.mmdb'
cache_size => 5000
source => '[@metadata][geoip]'
target => 'geoasn'
# worked in v.2.4: fields => ['asn','number']
fields => ['autonomous_system_number, autonomous_system_organization']
add_field => [ '[geoasn][asn]', '%{[geoasn][autonomous_system_organization]}' ]
add_field => [ '[geoasn][number]', '%{[geoasn][autonomous_system_number]}' ]
remove_field => ['[geoasn][autonomous_system_number]', '[geoasn][autonomous_system_organization]']
}
}
}