Hi Community,
Below is the logstash config . My objective is to get city,country,state based on geolocation. In input file i am putting geolocation.
input {
beats {
port=> 5044
#codec => json
}
}
filter {
geoip {
source => "[Geolocation]"
database => "/home/aniket/GeoLite2-City.mmdb"
target => "geoip"
fields => ["city_name", "region_name", "country_name"]
}
}
output
{
file {
path => "/home/aniket/test.json"
#codec => json_line
}
}
I have started the logstash in debug and below is the debug logs
[[main]-pipeline-manager] databasemanager - GeoIP database path is configured manually so the plugin will not check for update. Keep in mind that if you are not using the database shipped with this plugin, please go to https://www.maxmind.com/en/geolite2/eula and understand the terms and conditions.
[INFO ] 2024-07-16 20:15:06.488 [[main]-pipeline-manager] geoip - Using geoip database {:path=>"/home/aniket/GeoLite2-City.mmdb"}
[INFO ] 2024-07-16 20:15:06.635 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/etc/logstash/conf.d/geolocation.conf"], :thread=>"#<Thread:0x77fcb6af /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[INFO ] 2024-07-16 20:15:07.364 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.73}
[INFO ] 2024-07-16 20:15:07.375 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2024-07-16 20:15:07.380 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[DEBUG] 2024-07-16 20:15:07.390 [Converge PipelineAction::Create<main>] javapipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x77fcb6af /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[DEBUG] 2024-07-16 20:15:07.390 [logstash-pipeline-flush] PeriodicFlush - Pushing flush onto pipeline.
964 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Received a new payload
[DEBUG] 2024-07-16 20:15:48.968 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 1
[DEBUG] 2024-07-16 20:15:49.030 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 2
[DEBUG] 2024-07-16 20:15:49.032 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 3
[DEBUG] 2024-07-16 20:15:49.033 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 4
[DEBUG] 2024-07-16 20:15:49.034 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 5
[DEBUG] 2024-07-16 20:15:49.037 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 6
[DEBUG] 2024-07-16 20:15:49.040 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 7
[DEBUG] 2024-07-16 20:15:49.042 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 8
[DEBUG] 2024-07-16 20:15:49.044 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 9
[DEBUG] 2024-07-16 20:15:49.046 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 10
[DEBUG] 2024-07-16 20:15:49.051 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 11
[DEBUG] 2024-07-16 20:15:49.053 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 12
[DEBUG] 2024-07-16 20:15:49.055 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 13
[DEBUG] 2024-07-16 20:15:49.056 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:55022] Sending a new message for the listener, sequence: 14
[DEBUG] 2024-07-16 20:15:49.060 [defaultEventExecutorGroup-4-1] BeatsHandler - 21406b51: batches pending: false
[DEBUG] 2024-07-16 20:15:49.134 [[main]>worker9] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x6646f875>}
[DEBUG] 2024-07-16 20:15:49.135 [[main]>worker4] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x4cd9e42a>}
[DEBUG] 2024-07-16 20:15:49.137 [[main]>worker2] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x7252cef>}
[DEBUG] 2024-07-16 20:15:49.140 [[main]>worker10] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x3e7cac7c>}
[DEBUG] 2024-07-16 20:15:49.143 [[main]>worker5] geoip - IP was not found in the database {:event=>#<LogStash::Event:0xe6f3e20>}
[DEBUG] 2024-07-16 20:15:49.144 [[main]>worker7] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x35b688ae>}
[DEBUG] 2024-07-16 20:15:49.146 [[main]>worker3] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x6a9246aa>}
[DEBUG] 2024-07-16 20:15:49.152 [[main]>worker8] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x477f2f94>}
[DEBUG] 2024-07-16 20:15:49.153 [[main]>worker6] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x48b3ea96>}
[DEBUG] 2024-07-16 20:15:49.156 [[main]>worker1] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x2d691b0b>}
[DEBUG] 2024-07-16 20:15:49.157 [[main]>worker11] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x66582b30>}
[DEBUG] 2024-07-16 20:15:49.158 [[main]>worker11] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x4d164ff4>}
[DEBUG] 2024-07-16 20:15:49.158 [[main]>worker0] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x5c8ed6db>}
[DEBUG] 2024-07-16 20:15:49.160 [[main]>worker0] geoip - IP was not found in the database {:event=>#<LogStash::Event:0x2e27351c>}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker10] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker1] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker0] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.214 [[main]>worker8] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker6] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker0] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker1] file - Required path was deleted, creating the file again {:path=>"/home/aniket/test.json"}
[DEBUG] 2024-07-16 20:15:49.215 [[main]>worker2] file - File, writing event to file. {:filename=>"/home/aniket/test.json"}
[INFO ] 2024-07-16 20:15:49.215 [[main]>worker1] file - Opening file {:path=>"/home/aniket/test.json"}
In the test.json i am getting geoip lookup failure