# Logstash drop unnecessary fields ( ip2location plugin ) **URL:** \ **Category:** Logstash\ **Created:** [December 27, 2020, 6:49pm UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694 "2020-12-27T18:49:59Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Mohammad\_Mousavi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad_mousavi/32/56508_2.png) [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Post date:** [December 27, 2020, 6:49pm UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/1 "2020-12-27T18:49:59Z")
Hi, We use ip2locaion database and plugin to detect client's ISP. There are multiple versions of this database with different information but we bought the basic version with only ISP info. How can I prevent logstash to insert this "Not Supported" fields? ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cbb7f7ef5184808b67ef4c15e7a42a32e136da6e.png) This is my logstash config: ``` ip2location { source => "clientip" database => "/etc/logstash/conf.d/IP-COUNTRY-ISP.BIN" } ``` Thanx a lot. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 27, 2020, 7:14pm UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/2 "2020-12-27T19:14:45Z")
Why use ip2location rather than the elastic geoip filter? If you want to delete the unsupported fields you could do something like ``` ruby { code => ' l = event.get("ip2location") if l l.each { |k, v| if v == "Not_Supported" event.remove("[ip2location][#{k}]") end } end ' } ``` ---
**Author:** ![Mohammad\_Mousavi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad_mousavi/32/56508_2.png) [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Post date:** [December 28, 2020, 6:54am UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/3 "2020-12-28T06:54:27Z")
Thanx for help. As far as I know geoip in logstash or elasticsearch doesn't have ISP info. I'm using it already and it returns info like city and location not ISP. I did a lot of search and I came up with commercial databases like ip2location. ISP databases are not free on internet. There is some but they are too old. ---
**Author:** ![Mohammad\_Mousavi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad_mousavi/32/56508_2.png) [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Post date:** [January 9, 2021, 9:07am UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/4 "2021-01-09T09:07:08Z")
This filter causes logstash crash. This is the log: ``` [2021-01-09T12:33:41,129][ERROR][org.logstash.execution.WorkerLoop][main] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. java.util.ConcurrentModificationException: null at java.util.IdentityHashMap.forEach(IdentityHashMap.java:1358) ~[?:?] at org.logstash.Rubyfier.deepMap(Rubyfier.java:71) ~[logstash-core.jar:?] at org.logstash.Rubyfier.lambda$initConverters$8(Rubyfier.java:99) ~[logstash-core.jar:?] at org.logstash.Rubyfier.deep(Rubyfier.java:53) ~[logstash-core.jar:?] at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_get_field(JrubyEventExtLibrary.java:85) ~[logstash-core.jar:?] at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_1_dot_4.lib.logstash.filters.ip2location.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ip2location-2.1.4/lib/logstash/filters/ip2location.rb:50) ~[?:?] at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:253) ~[jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?] at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:253) ~[jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162) ~[?:?] at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.BlockBody.yield(BlockBody.java:114) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.Block.yield(Block.java:170) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyArray.each(RubyArray.java:1800) ~[jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?] at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) ~[jruby-complete-9.2.8.0.jar:?] at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:99) ~[logstash-core.jar:?] at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:115) ~[logstash-core.jar:?] at org.logstash.generated.CompiledDataset117.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset118.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset119.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset120.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset121.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset122.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset123.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset124.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset125.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset126.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset127.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset128.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset129.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset130.compute(Unknown Source) ~[?:?] at org.logstash.generated.CompiledDataset131.compute(Unknown Source) ~[?:?] at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.8.0.jar:?] at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.8.0.jar:?] at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.8.0.jar:?] at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?] at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?] at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?] at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?] at java.lang.Thread.run(Thread.java:834) [?:?] [2021-01-09T12:33:41,325][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.util.ConcurrentModificationException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]} [2021-01-09T12:33:41,374][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 9, 2021, 3:08pm UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/5 "2021-01-09T15:08:18Z")
> [@Badger](#): > > `if v == "Not_Supported"` Does it help if you change that to ``` if v.is_a? "String" and v == "Not_Supported" ``` ---
**Author:** ![GaryChewter](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@GaryChewter](https://discuss.elastic.co/u/GaryChewter)\ **Post date:** [January 18, 2021, 4:14am UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/6 "2021-01-18T04:14:22Z")
You can add the following codes into your filter section, ```auto mutate { remove_field => ["[ip2location][area_code]", "[ip2location][city]", "[ip2location][domain]", ... ] } ``` Note: `...` represents other fields that you would like to remove ---
**Author:** ![Mohammad\_Mousavi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammad_mousavi/32/56508_2.png) [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Post date:** [January 18, 2021, 11:42am UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/7 "2021-01-18T11:42:46Z")
I did it. Still the same. it crashes on removing one of them. Every time on different one. When I comment the whole filter, it does not crash. I think some thing is wrong with this removing. ``` [2021-01-18T15:21:38,302][DEBUG][logstash.filters.mutate][main] filters/LogStash::Filters::Mutate: removing field {:field=>"[ip2location][zip_code]"} [2021-01-18T15:21:38,417][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.util.ConcurrentModificationException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:834)"]} [2021-01-18T15:21:38,470][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit ``` Even I add ` if [ip2location][isp]` before remove but still the same. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 15, 2021, 11:43am UTC](https://discuss.elastic.co/t/logstash-drop-unnecessary-fields-ip2location-plugin/259694/8 "2021-02-15T11:43:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.