I got geoip enrichment working with packetbeat a few days ago in a test set up with elasticsearch 8.5 and kibana 8.5. But I tore down the servers this morning so that I can repeat the installation. For some reason, this time I can't get the geoip enrichment to work. After I installed the elk stack and packetbeat, I ran this command in the Dev Tools of kibana (which I took from this documentation Enrich events with geoIP information | Packetbeat Reference [master] | Elastic)
PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{
"geoip": {
"field": "client.ip",
"target_field": "client.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "client.ip",
"target_field": "client.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "source.ip",
"target_field": "source.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "source.ip",
"target_field": "source.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "destination.ip",
"target_field": "destination.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "destination.ip",
"target_field": "destination.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "server.ip",
"target_field": "server.geo",
"ignore_missing": true
}
},
{
"geoip": {
"database_file": "GeoLite2-ASN.mmdb",
"field": "server.ip",
"target_field": "server.as",
"properties": [
"asn",
"organization_name"
],
"ignore_missing": true
}
},
{
"geoip": {
"field": "host.ip",
"target_field": "host.geo",
"ignore_missing": true
}
},
{
"rename": {
"field": "server.as.asn",
"target_field": "server.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "server.as.organization_name",
"target_field": "server.as.organization.name",
"ignore_missing": true
}
},
{
"rename": {
"field": "client.as.asn",
"target_field": "client.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "client.as.organization_name",
"target_field": "client.as.organization.name",
"ignore_missing": true
}
},
{
"rename": {
"field": "source.as.asn",
"target_field": "source.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "source.as.organization_name",
"target_field": "source.as.organization.name",
"ignore_missing": true
}
},
{
"rename": {
"field": "destination.as.asn",
"target_field": "destination.as.number",
"ignore_missing": true
}
},
{
"rename": {
"field": "destination.as.organization_name",
"target_field": "destination.as.organization.name",
"ignore_missing": true
}
}
]
}
Then I added output.elasticsearch.pipeline: geoip-info
to my /etc/packetbeat/packetbeat.yml
and systemctl restart packetbeat
.
But when I look in Discover
for kibana, I don't see the field for client.geo.location
. I stopped the packetbeat service with systemctl stop packetbeat
. Then I deleted the packetbeat index through kibana, then I re-ran the packetbeat set up with /usr/share/packetbeat/bin/packetbeat setup -c /etc/packetbeat/packetbeat.yml ...other flags...
. Then I did a systemctl start packetbeat.service
But still, the client.geo.location
doesn't appear in the in Discover
for kibana.
I also ran this command:
PUT my-index-000001/_doc/my_id?pipeline=geoip-info
{
"ip": "89.160.20.128"
}
Which showed success. Then I ran this command:
GET my-index-000001/_doc/my_id
Which gave this result:
{
"_index": "my-index-000001",
"_id": "my_id",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"found": true,
"_source": {
"ip": "89.160.20.128",
"tags": [
"_geoip_database_unavailable_GeoLite2-City.mmdb",
"_geoip_database_unavailable_GeoLite2-ASN.mmdb",
"_geoip_database_unavailable_GeoLite2-City.mmdb",
"_geoip_database_unavailable_GeoLite2-ASN.mmdb",
"_geoip_database_unavailable_GeoLite2-City.mmdb",
"_geoip_database_unavailable_GeoLite2-ASN.mmdb",
"_geoip_database_unavailable_GeoLite2-City.mmdb",
"_geoip_database_unavailable_GeoLite2-ASN.mmdb",
"_geoip_database_unavailable_GeoLite2-City.mmdb"
]
}
}
The last time I successfully got geo ip enrichment to work, I vaguely recall I did a GET
request to an elastic endpoint which then gave a response saying I successfully downloaded some geoip data. But I can't remember exactly what that command was and I can't remember where that documentation for this command is. Or maybe I did something else, I can't remember
Does anyone know what I did wrong this time? What can I do to get geoip to work?