I have Elasticsearch version 8.15.1 and filebat 8.16.1.
The filebeat receives logs from a fortigate, and after several dissects it sends to elastic, these logs are received in elastic but without the Geoip enrichment. It is not running the fortinet-geoip-pipeline which is configured in elastic and filebeat.
My filebeat.yml:
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
filebeat.inputs:
# Input para UDP
- type: udp
enabled: true
host: "0.0.0.0:9004" # input
timeout: 5s
tags: ["fortinet-fortigate", "fortinet-firewall"]
processors:
#... N dissects ....
- dissect:
when:
regexp:
fortinet.firewall.sub_log2: "^shost"
tokenizer: "shost=%{shost} ........ xxxx=%{remote} xxxx=%{local} ... #<-- prefix fortinet.firewall .remote and .local
field: "message"
target_prefix: "fortinet.firewall"
ignore_failure: true
#.... N dissects ....
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
host: "xxxxxxxx"
ssl.certificate_authorities: ["xxxxxx"]
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["https://xxxxx"]
ssl.certificate_authorities: ["xxxxx"]
username: "xxxxxx"
password: "${xxxxxx}"
pipeline: fortinet-geoip-pipeline # <--------------------------------- here
preset: balanced
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
My elasticsearch.yml:
... #other lines
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
ingest.geoip.downloader.enabled: true
ingest.geoip.downloader.eager.download: true
My pipeline:
{
"fortinet-geoip-pipeline": {
"description": "Add complete GeoIP information based on fortinet.firewall.remote",
"processors": [
{
"geoip": {
"field": "fortinet.firewall.remote",
"target_field": "geo",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"geoip": {
"field": "fortinet.firewall.local",
"target_field": "geo",
"ignore_missing": true,
"ignore_failure": true
}
}
]
}
}
if I execute POST _ingest/pipeline/fortinet-geoip-pipeline/_simulate?verbose=true With data that I have in the events, it returns me with the geoip of the corresponding fields. It works fine.
When looking at the number of executions in the pipeline I only see mine GET /_nodes/stats this is a correct response:
{
"docs": [
{
"processor_results": [
{
"processor_type": "geoip",
"status": "success",
"doc": {
"_index": "_index",
"_version": "-3",
"_id": "_id",
"_source": {
"geo": {
"continent_name": "South America",
"region_iso_code": "xxxx",
"city_name": "xxxx",
"country_iso_code": "xxxx",
"country_name": "xxxx",
"location": {
"lon": -00.000,
"lat": -000.000
},
"region_name": "xxxxx"
},
"input": {
"type": "udp"
},
"agent": {
.....
"fortinet": {
"firewall": {
"srcuuid": "39426",
"msg": "SSL user failed to logged in",
"reason": "sslvpn_login_unknown_user",
"remote": "xxx.xxx.xxx.xxxx",
GET _ingest/geoip/stats Returns ok without failures
{
"stats": {
"successful_downloads": 3,
"failed_downloads": 0,
"total_download_time": 11874,
"databases_count": 3,
"skipped_updates": 0,
"expired_databases": 0
},
"nodes": {
"k586sYdpRte5O_sDCZsEQg": {
"databases": [
{
"name": "GeoLite2-ASN.mmdb"
},
{
"name": "GeoLite2-City.mmdb"
},
{
"name": "GeoLite2-Country.mmdb"
}
],
"files_in_temp": [
"GeoLite2-Country.mmdb_LICENSE.txt",
"GeoLite2-ASN.mmdb",
"GeoLite2-ASN.mmdb_LICENSE.txt",
"GeoLite2-City.mmdb_LICENSE.txt",
"GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
"GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
"GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
"GeoLite2-ASN.mmdb_COPYRIGHT.txt",
"GeoLite2-City.mmdb_README.txt",
"GeoLite2-Country.mmdb_COPYRIGHT.txt",
"GeoLite2-City.mmdb",
"GeoLite2-Country.mmdb",
"GeoLite2-City.mmdb_COPYRIGHT.txt"
],
"cache_stats": {
"count": 0,
"hits": 0,
"misses": 0,
"evictions": 0,
"hits_time_in_millis": 0,
"misses_time_in_millis": 0
}
}
}
}
can you help me?