Hey there,
I'm currently running Packetbeat on a linux box and it sends directly data to an Elasticsearch cluster.
Everything runs perfectly fine, except when I want to add geoIP info following this documentation. Indeed, as soon as I update the config file, I don't receive any data anymore.
Events continue to be emitted, but nothing appears in the dashboard [Packetbeat] Overview ECS anymore. Fail rates climbs to 3/s and open handles climb to 10/s.
The weird thing is that when I test the config with sudo packetbeat test config -c packetbeat.yml -e , it says that the config is ok. And when I launch it as a service, it runs fine, but I still don't receive any data.
As soon as I take away the pipeline in the config file and restart the service, everything runs fine again and I receive all the data.
Do you have any idea what's happening and how I could fix this ?
In case you need it, here is the config file:
packetbeat.interfaces.device: any
packetbeat.flows:
timeout: 30s
period: 10s
packetbeat.protocols:
- type: icmp
enabled: true
- type: amqp
ports: [5672]
- type: cassandra
ports: [9042]
- type: dhcpv4
ports: [67, 68]
- type: dns
ports: [53]
- type: http
ports: [80, 8080, 8000, 5000, 8002]
- type: memcache
ports: [11211]
- type: mysql
ports: [3306,3307]
- type: pgsql
ports: [5432]
- type: redis
ports: [6379]
- type: thrift
ports: [9090]
- type: mongodb
ports: [27017]
- type: nfs
ports: [2049]
- type: tls
ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
name: "box_name"
setup.template.pattern: "packetbeat*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 2
index.number_of_replicas: 1
index.routing.allocation.require.temp: "hot"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "packetbeat"
setup.ilm.pattern: "000001"
setup.ilm.policy_name: "hotwarm_policy"
tags: ["my_tags"]
setup.kibana:
host: "kibana_ip:5601"
username: "${KIB_USER}"
password: "${KIB_PWD}"
output.elasticsearch:
hosts: ["elasticsearch_node1_ip:9200","elasticsearch_node2_ip:9200"]
username: "${ES_USER}"
password: "${ES_PWD}"
pipeline: geoip-info
processors:
- add_host_metadata:
netinfo.enabled: true
Geo:
name: box_name
location: xxxxx
continent_name: xxxxx
country_iso_code: xxxxx
region_name: xxxxx
region_iso_code: xxxxx
city_name: xxxxx
- add_locale: ~
- add_cloud_metadata: ~
- add_fields:
when.network.source.ip: private
fields:
source.geo.location:
lat: xxxxx
lon: xxxxx
source.geo.continent_name: xxxxx
source.geo.region_iso_code: xxxxx
source.geo.country_iso_code: xxxxx
source.geo.region_name: xxxxx
source.geo.name: box_name
target: ''
- add_fields:
when.network.destination.ip: private
fields:
source.geo.location:
lat: xxxxx
lon: xxxxx
source.geo.continent_name: xxxxx
source.geo.region_iso_code: xxxxx
source.geo.country_iso_code: xxxxx
source.geo.region_name: xxxxx
source.geo.name: xxxxx
target: ''
monitoring.enabled: true
and here is the pipeline in the elasticsearch cluster:
"geoip-info" : {
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "client.ip",
"target_field" : "client.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "source.ip",
"target_field" : "source.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "destination.ip",
"target_field" : "destination.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "server.ip",
"target_field" : "server.geo",
"ignore_missing" : true
}
},
{
"geoip" : {
"field" : "host.ip",
"target_field" : "host.geo",
"ignore_missing" : true
}
}
]
}
Thanks in advance for your help !
Cheers,
jsu