Geoip - Apache logs

Elastic Stack Logstash
1

Hi,

I'm trying to get geoip info from ipclient from my access logs generated by a apache2 server.

I tried this filter in logstash:

filter {

        if  [tags][path] =~ "/var/log/apache2/xxx-access.log" {
                geoip {
                        source => "clientip"
                }
        }

}

Unfortunately I get errors once I restart Logstash :

avril 04 04:55:47 debian logstash[11337]: [2021-04-04T04:55:47,340][ERROR][logstash.javapipeline ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"class java.util.HashMap cannot be cast to class java.lang.String (java.util.HashMap and java.lang.String are in module java.base of loader 'bootstrap')", :exception=>Java::JavaLang::ClassCastException, :backtrace=>["org.logstash.Event.existingTag(Event.java:430)", "org.logstash.Event.tag(Event.java:406)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_tag(JrubyEventExtLibrary.java:277)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$1$0$ruby_tag.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$1$0$ruby_tag.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:833)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_geoip_minus_6_dot_0_dot_5_minus_java.lib.logstash.filters.geoip.RUBY$block$tag_unsuccessful_lookup$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.5-java/lib/logstash/filters/geoip.rb:118)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:106)", "org.jruby.runtime.Block.yield(Block.java:184)", "org.jruby.RubyArray.each(RubyArray.java:1809)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_geoip_minus_6_dot_0_dot_5_minus_java.lib.logstash.filters.geoip.RUBY$method$tag_unsuccessful_lookup$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.5-java/lib/logstash/filters/geoip.rb:118)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_geoip_minus_6_dot_0_dot_5_minus_java.lib.logstash.filters.geoip.RUBY$method$tag_unsuccessful_lookup$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.5-java/lib/logstash/filters/geoip.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_geoip_minus_6_dot_0_dot_5_minus_java.lib.logstash.filters.geoip.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.5-java/lib/logstash/filters/geoip.rb:112)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:386)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:184)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:338)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:197)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:106)", "org.jruby.runtime.Block.yield(Block.java:184)", "org.jruby.RubyArray.each(RubyArray.java:1809)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:127)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:134)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:329)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:323)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base/java.lang.reflect.Method.invoke(Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"], :thread=>"#<Thread:0xed404d1 sleep>"}

Does anyone have any idea of my mistake ? Im new to ElasticStack :slight_smile:

Thanks

2

You are getting the error class java.util.HashMap cannot be cast to class java.lang.String. From that if it appears that you have a [tags] field that is a hash, but [tags] is a field that logstash assumes is an array, which is not compatible with it being a hash, so when it calls the event.tag method it gets an exception.

3

So this condition should work ?

filter {

        if "/var/log/apache2/xxx-access.log" in [tags] {
                geoip {
                        source => "clientip"
                }
        }

}
4

[tags] has to be an array, it cannot be a hash. If you add that path to the array then that condition should work.

5

In fact my data comes like this with telegraf:

{
  "_index": "syslog-2021-04-04",
  "_type": "_doc",
  "_id": "zRMknngZEi4BvIimBwCO",
  "_score": 1,
  "_source": {
    "port": 56170,
    "fields": {
      "httpversion": "1.1",
      "verb": "GET",
      "clientip": "x.x.x.x",
      "timestamp": "04/Apr/2021:17:15:13 +0000",
      "request": "/wp-login.php",
      "ident": "-",
      "response": "404",
      "bytes": "436",
      "auth": "-"
    },
    "tags": {
      "path": "/var/log/apache2/xxx-access.log",
      "host": "raspberry"
    },
    "timestamp": 1617556513,
    "name": "syslog_files",
    "host": "ubuntu.home",
    "@version": "1",
    "@timestamp": "2021-04-04T18:27:43.498Z"
  },
  "fields": {
    "@timestamp": [
      "2021-04-04T18:27:43.498Z"
    ]
  }
}

And I don't understand how I should do in order to set fields.clientip as the source for geoip.

I tried multiple configurations but no one worked for me.
I folowed this part of the doc https://www.elastic.co/guide/en/logstash/7.12/event-dependent-configuration.html#sprintf

I tried things such as "[fields][clientip]" or "%{"[fields][clientip]}" etc..

6

Use

geoip { source => "[fields][clientip]" }

As I said, [tags] cannot be a hash. I suggest you use mutate+rename to rename it.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.