Logstash drop unnecessary fields ( ip2location plugin )

Mohammad_Mousavi · December 27, 2020, 6:49pm

Hi, We use ip2locaion database and plugin to detect client's ISP.
There are multiple versions of this database with different information but we bought the basic version with only ISP info.
How can I prevent logstash to insert this "Not Supported" fields?

This is my logstash config:

ip2location {
source => "clientip"
database => "/etc/logstash/conf.d/IP-COUNTRY-ISP.BIN"
   }

Thanx a lot.

Badger · December 28, 2020, 6:48am

Why use ip2location rather than the elastic geoip filter?

If you want to delete the unsupported fields you could do something like

ruby {
    code => '
        l = event.get("ip2location")
        if l
            l.each { |k, v|
                if v == "Not_Supported"
                    event.remove("[ip2location][#{k}]")
                end
            }
        end
    '
}

Mohammad_Mousavi · December 28, 2020, 6:54am

Thanx for help.
As far as I know geoip in logstash or elasticsearch doesn't have ISP info. I'm using it already and it returns info like city and location not ISP. I did a lot of search and I came up with commercial databases like ip2location. ISP databases are not free on internet. There is some but they are too old.

Mohammad_Mousavi · January 9, 2021, 9:07am

This filter causes logstash crash. This is the log:

[2021-01-09T12:33:41,129][ERROR][org.logstash.execution.WorkerLoop][main] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
java.util.ConcurrentModificationException: null
        at java.util.IdentityHashMap.forEach(IdentityHashMap.java:1358) ~[?:?]
        at org.logstash.Rubyfier.deepMap(Rubyfier.java:71) ~[logstash-core.jar:?]
        at org.logstash.Rubyfier.lambda$initConverters$8(Rubyfier.java:99) ~[logstash-core.jar:?]
        at org.logstash.Rubyfier.deep(Rubyfier.java:53) ~[logstash-core.jar:?]
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_get_field(JrubyEventExtLibrary.java:85) ~[logstash-core.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_1_dot_4.lib.logstash.filters.ip2location.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ip2location-2.1.4/lib/logstash/filters/ip2location.rb:50) ~[?:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:253) ~[jruby-complete-9.2.8.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:253) ~[jruby-complete-9.2.8.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162) ~[?:?]
        at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.runtime.BlockBody.yield(BlockBody.java:114) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.runtime.Block.yield(Block.java:170) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.RubyArray.each(RubyArray.java:1800) ~[jruby-complete-9.2.8.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?]
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) ~[jruby-complete-9.2.8.0.jar:?]
        at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:99) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:115) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset117.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset118.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset119.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset120.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset121.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset122.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset123.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset124.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset125.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset126.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset127.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset128.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset129.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset130.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset131.compute(Unknown Source) ~[?:?]
        at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.8.0.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-01-09T12:33:41,325][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.util.ConcurrentModificationException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}
[2021-01-09T12:33:41,374][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Badger · January 9, 2021, 3:08pm

Does it help if you change that to

if v.is_a? "String" and v == "Not_Supported"

GaryChewter · January 18, 2021, 4:14am

You can add the following codes into your filter section,

mutate {
    remove_field => [ "[ip2location][area_code]", "[ip2location][city]", "[ip2location][domain]", ... ]
}

Note: ... represents other fields that you would like to remove

Mohammad_Mousavi · January 18, 2021, 12:09pm

I did it. Still the same. it crashes on removing one of them. Every time on different one.
When I comment the whole filter, it does not crash. I think some thing is wrong with this removing.

[2021-01-18T15:21:38,302][DEBUG][logstash.filters.mutate  ][main] filters/LogStash::Filters::Mutate: removing field {:field=>"[ip2location][zip_code]"}
[2021-01-18T15:21:38,417][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.util.ConcurrentModificationException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]}
[2021-01-18T15:21:38,470][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Even I add if [ip2location][isp] before remove but still the same.

system · February 15, 2021, 11:43am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.