Logstash geoip failure | IP was not found in the database

Elastic Stack Logstash
1

Hi Community,

Below is the logstash config . My objective is to get city,country,state based on geolocation. In input file i am putting geolocation.

input {
beats {
port=> 5044
#codec => json
}
}
filter {
 geoip {
    source => "[Geolocation]"
    database => "/home/aniket/GeoLite2-City.mmdb"
    target => "geoip"
fields => ["city_name", "region_name", "country_name"]
  }
}

output
{
file {
path => "/home/aniket/test.json"
#codec => json_line
}
}

I have started the logstash in debug and below is the debug logs

[[main]-pipeline-manager] databasemanager - GeoIP database path is configured manually so the plugin will not check for update. Keep in mind that if you are not using the database shipped with this plugin, please go to https://www.maxmind.com/en/geolite2/eula and understand the terms and conditions.
[INFO ] 2024-07-16 20:15:06.488 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/home/aniket/GeoLite2-City.mmdb"}
[INFO ] 2024-07-16 20:15:06.635 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/etc/logstash/conf.d/geolocation.conf"], :thread=>"#<Thread:0x77fcb6af /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[INFO ] 2024-07-16 20:15:07.364 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.73}
[INFO ] 2024-07-16 20:15:07.375 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2024-07-16 20:15:07.380 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[DEBUG] 2024-07-16 20:15:07.390 [Converge PipelineAction::Create<main>] javapipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x77fcb6af /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[DEBUG] 2024-07-16 20:15:07.390 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
964 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Received a new payload
[DEBUG] 2024-07-16 20:15:48.968 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 1
[DEBUG] 2024-07-16 20:15:49.030 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 2
[DEBUG] 2024-07-16 20:15:49.032 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 3
[DEBUG] 2024-07-16 20:15:49.033 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 4
[DEBUG] 2024-07-16 20:15:49.034 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 5
[DEBUG] 2024-07-16 20:15:49.037 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 6
[DEBUG] 2024-07-16 20:15:49.040 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 7
[DEBUG] 2024-07-16 20:15:49.042 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 8
[DEBUG] 2024-07-16 20:15:49.044 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 9
[DEBUG] 2024-07-16 20:15:49.046 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 10
[DEBUG] 2024-07-16 20:15:49.051 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 11
[DEBUG] 2024-07-16 20:15:49.053 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 12
[DEBUG] 2024-07-16 20:15:49.055 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 13
[DEBUG] 2024-07-16 20:15:49.056 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 14
[DEBUG] 2024-07-16 20:15:49.060 [defaultEventExecutorGroup-4-1] BeatsHandler - 21406b51: batches pending: false
[DEBUG] 2024-07-16 20:15:49.134 [[main]>worker9] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x6646f875>}
[DEBUG] 2024-07-16 20:15:49.135 [[main]>worker4] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x4cd9e42a>}
[DEBUG] 2024-07-16 20:15:49.137 [[main]>worker2] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x7252cef>}
[DEBUG] 2024-07-16 20:15:49.140 [[main]>worker10] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x3e7cac7c>}
[DEBUG] 2024-07-16 20:15:49.143 [[main]>worker5] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0xe6f3e20>}
[DEBUG] 2024-07-16 20:15:49.144 [[main]>worker7] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x35b688ae>}
[DEBUG] 2024-07-16 20:15:49.146 [[main]>worker3] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x6a9246aa>}
[DEBUG] 2024-07-16 20:15:49.152 [[main]>worker8] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x477f2f94>}
[DEBUG] 2024-07-16 20:15:49.153 [[main]>worker6] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x48b3ea96>}
[DEBUG] 2024-07-16 20:15:49.156 [[main]>worker1] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x2d691b0b>}
[DEBUG] 2024-07-16 20:15:49.157 [[main]>worker11] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x66582b30>}
[DEBUG] 2024-07-16 20:15:49.158 [[main]>worker11] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x4d164ff4>}
[DEBUG] 2024-07-16 20:15:49.158 [[main]>worker0] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x5c8ed6db>}
[DEBUG] 2024-07-16 20:15:49.160 [[main]>worker0] geoip - IP  was not found in the database {:event=>#<LogStash::Event:0x2e27351c>}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker10] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker1] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker0] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker8] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker6] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker0] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker1] file - Required path was deleted, creating the file again {:path=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker2] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[INFO ] 2024-07-16 20:15:49.215 [[main]>worker1] file - Opening file {:path=>"/home/aniket/test.json"}

In the test.json i am getting geoip lookup failure

2

Hello,

Please share some sample documents with some IP addresses that you have the failure tag so people can try to replicate the issue.

3

Thanks for your quick response @leandrojmp . Below is the sample logs.

[
{
  "Name": "Aniket",
  "Age": 29,
  "Geolocation": "POINT(28.6139,77.2090)",
  "Timestamp": "2024-07-16T18:15:00+05:30"
},
{
  "Name": "Arun Kumar",
  "Age": 30,
  "Geolocation": "POINT(12.9716,77.5946)",
  "Timestamp": "2024-07-16T18:30:00+05:30"
}
]

Sorry i haven't written properly in the logs i am not getting IP address i am getting only geolocations.

4

Not sure what you want to do, but the geoip filter requires an ip address in the source field, it will then try to match this ip address in its database and if it has a match, it will then populate the target field with geo locaton information.

If you do not have an ip address to search for the geo location information, then the geoip filter will not work.

It seems that you want to do the reverse operation, find an ip address using a geolocation, this is not possible with any filter in logstash.

5

This is referred to as reverse geocoding. logstash does not support it, but there are several third-party APIs you could call.

6

I have checked this case but I don't know what are the steps to get the required data

7

So that means we can't use any elasticsearch enrich policy to get city,state,country using lat,long ?

8

In a native way no, you can only get geo location information using the IP address, not the inverse.

What you can do is to get that data elsewhere and store it in Elasticsearch, then you can use an enrich policy to enrich your indices.

The linked post mentiones an example on how to do that.