GeoIP issue on logstash conf file

Elastic Stack Logstash
1 
[ERROR][logstash.filters.geoip   ] Invalid setting for geoip filter plugin:

  filter {
    geoip {
      # This setting must be a path
      # File does not exist or cannot be opened ./usr/share/logstash/GeoLite2-City.mmdb
      database => "./usr/share/logstash/GeoLite2-City.mmdb"
      ...
    }
  }

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:testing, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}

using latest version of logstash 8.12.0
Not able to connect logstash pipeline with kafka 3 pipelines are facing the same error in which geo ip setting is there.
I have given permission as well
sudo chown -R logstash:root /usr/share/logstash/pipeline/GeoLite2-ASN.mmdb && sudo chmod -R g+rw /usr/share/logstash/pipeline/GeoLite2-ASN.mmdb

2

This seems wrong, there is a dot in front of the path.

3

I have removed the . and it is still not working

4

What is the error now? If you removed the . then the error must changed, you need to share the logs.

5

Are you using the correct path?

In your configuration you have the City database in /usr/share/logstash/, but then you said that you give permissions to the ASN database in /usr/share/logstash/pipeline/.

Double check your paths.

Also, any reason to not use the built-in databases? Is it an air-gapped system?

6

basically i was moving data from my on going cluster to a new cluster with latest version.
so i copied all the data as it is to the new server from the old server.
i copied it in /usr/share/logstash/pipeline.
all the three files of geoip are under this directory.
and the pipeline which is facing error is testing /usr/share/logstash/pipeline/testing

7

and these geo ip files are working perfectly fine on my ongoing cluster.
it is just that the new one with latest version is facing problem.
and i tried removing the . but the error is same then it says that /usr/share/logstash/pipeline/Geo cannot be opened or accessed

8

The error you shared is related to Logstash not being able to find the database.

In your config you are pointing to /usr/share/logstash/database-file, but then you shared that you changed permissions to a different database type on /usr/share/logstash/pipeline.

You need to double check the paths in your configuration, you need to point to the exactly location that the database file is.

9

Yeah, but there is any reason to download yourself the database instead of letting Logstash manage this?

If you do not have any specific reason for that and your Logstash server has internet access to download and update the database, you may remove this line, as explained in the documentation, Logstash can download and update the maxmind geolite database by itself.

10 
[2024-02-25T18:59:53,528][ERROR][logstash.javapipeline    ][testing] Pipeline error {:pipeline_id=>"testing", :exception=>#<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity mode requires a `target` when `source` is not an `ip` sub-field, eg. [client][ip]>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-geoip-7.2.13-java/lib/logstash/filters/geoip.rb:143:in `auto_target_from_source!'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-geoip-7.2.13-java/lib/logstash/filters/geoip.rb:133:in `setup_target_field'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-geoip-7.2.13-java/lib/logstash/filters/geoip.rb:108:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1989:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/testing/testing-filter-tag.conf", "/usr/share/logstash/pipeline/testing/testing-filter.conf", "/usr/share/logstash/pipeline/testing/testing-input.conf", "/usr/share/logstash/pipeline/testing/testing-output.conf"], :thread=>"#<Thread:0x47de9ea0 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

[ERROR][logstash.agent           ] Failed to execute action {:id=>:testing, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<testing>, action_result: false", :backtrace=>nil}
11 
 sudo chown -R logstash:root /usr/share/logstash/GeoLite2-City.mmdb && sudo chmod -R g+rw /usr/share/logstash/GeoLite2-City.mmdb
sudo chown -R logstash:root /usr/share/logstash/GeoLite2-Country.mmdb && sudo chmod -R g+rw /usr/share/logstash/GeoLite2-Country.mmdb
 sudo chown -R logstash:root /usr/share/logstash/GeoLite2-ASN.mmdb && sudo chmod -R g+rw /usr/share/logstash/GeoLite2-ASN.mmdb

I removed the . as suggested by you and gave the right path and used the right ownership command

12

What does your geoip filter looks like?

You didn't share your entire filter, nor you said which version are you using.

It seems that you are using version 8. Did you migrate from version 7?

There are breaking changes when migrating from 7 to 8 that will require changes in your configuration, some of them are related to the ecs compatibility mode.

Add this setting to your geoip filter.

ecs_compatibility => disabled
13

No i was just normally taking the backup from the ongoing cluster.
There is no specific reason for it.
The filter files were way too long so i didn't want to go to each pipeline and change the lines for Geoip.
So i simply copied them and tried running the file.

14

Yes I am using version 8.12.
No i didn't migrate.
I made a new setup.
what kind of changes should be there in the latest version for GeoIP?
As I was copying the pipelines from my ongoing cluster which has version 7.

15

So you migrate from 7 to 8, there are breaking changes that you need to always read and see if they will impact you.

In your case and according to the log you shared is this one.

The ECS compatibility mode is enabled by default on version 8, it was disabled on version 7, you need to disable it using the setting mentioned in the previous answer.

16 
ecs_compatibility => disabled

where should add this line in the output file or input file or filter file where there are lines for GeoIP?

17

As mentioned you need to add this inside your geoip filter, it is a configuration option as mentioned in the documentation.

This will disable the the ecs compatiblity and the error you are receiving now should go away.

18 
#if ![geoip] and [src_ip] !~ /^(1\.|1\.1\.)/ {
    #if [host] =~ /1\.0\.0\.2/ {

    #geoip {
        #add_tag => [ "GeoIP" ]
        #source => "src_ip"
        #add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        #add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
        #database => "/usr/share/logstash/GeoLite2-ASN.mmdb"

    #}
    if ![geoip] and [src_ip] !~ /^(1\.|1\.1\.)/ {
        geoip {
                #add_tag => [ "GeoIP" ]
                source => "src_ip"
    database => "/usr/share/logstash/GeoLite2-City.mmdb"
              }
    }

under testing-filter.conf i found these lines related to GeoIP.
So in each pipeline i have to go to the filter files and search for GeoIP and then add the ecs compatibility line?
Nothing should be done in logstash.ym / piplines.ym /input.conf /output.conf just the filter file?

19

If you have multiple geoip filters in multiple pipelines, it will be easier to disable the ecs compatiblity globally, this can be done adding a setting in logstash.yml.

If you add the following line in your logstash.yml and restart logstash, it will disable the ecs compatiblity in all filters in all pipelines.

pipeline.ecs_compatibility: disabled

This is mentioned in the breaking changes documentation, please read the documentations that are linked.

You can disable the ecs compatibility per filter, per pipeline or globally, since you are migrating from 7 to 8 and didn't checked the breaking changes yet, I would say that it would be easier to disable it globally.

20

Thankyou so much now there is no such error.