Geoip not work with filebeat

I have Elasticsearch version 8.15.1 and filebat 8.16.1.

The filebeat receives logs from a fortigate, and after several dissects it sends to elastic, these logs are received in elastic but without the Geoip enrichment. It is not running the fortinet-geoip-pipeline which is configured in elastic and filebeat.

My filebeat.yml:

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: false
  # Period on which files under path should be checked for changes
  #reload.period: 10s

filebeat.inputs:
  # Input para UDP
  - type: udp
    enabled: true
    host: "0.0.0.0:9004"  # input
    timeout: 5s
    tags: ["fortinet-fortigate", "fortinet-firewall"]
    processors:
	
	#... N dissects ....
	
      - dissect:
          when:
            regexp:
              fortinet.firewall.sub_log2: "^shost"
          tokenizer: "shost=%{shost} ........ xxxx=%{remote} xxxx=%{local} ...  #<-- prefix fortinet.firewall .remote and .local
          field: "message"
          target_prefix: "fortinet.firewall"
          ignore_failure: true
	#.... N dissects ....

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  host: "xxxxxxxx"
  ssl.certificate_authorities: ["xxxxxx"]

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://xxxxx"]
  ssl.certificate_authorities: ["xxxxx"]
  username: "xxxxxx"    
  password: "${xxxxxx}"
  pipeline: fortinet-geoip-pipeline  # <--------------------------------- here
  preset: balanced


# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

My elasticsearch.yml:


... #other lines

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
ingest.geoip.downloader.enabled: true
ingest.geoip.downloader.eager.download: true

My pipeline:

{
  "fortinet-geoip-pipeline": {
    "description": "Add complete GeoIP information based on fortinet.firewall.remote",
    "processors": [
      {
        "geoip": {
          "field": "fortinet.firewall.remote",
          "target_field": "geo",
          "ignore_missing": true,
          "ignore_failure": true
        }
      },
      {
        "geoip": {
          "field": "fortinet.firewall.local",
          "target_field": "geo",
          "ignore_missing": true,
          "ignore_failure": true
        }
      }
    ]
  }
}


if I execute POST _ingest/pipeline/fortinet-geoip-pipeline/_simulate?verbose=true With data that I have in the events, it returns me with the geoip of the corresponding fields. It works fine.

When looking at the number of executions in the pipeline I only see mine GET /_nodes/stats this is a correct response:

{
  "docs": [
    {
      "processor_results": [
        {
          "processor_type": "geoip",
          "status": "success",
          "doc": {
            "_index": "_index",
            "_version": "-3",
            "_id": "_id",
            "_source": {
              "geo": {
                "continent_name": "South America",
                "region_iso_code": "xxxx",
                "city_name": "xxxx",
                "country_iso_code": "xxxx",
                "country_name": "xxxx",
                "location": {
                  "lon": -00.000,
                  "lat": -000.000
                },
                "region_name": "xxxxx"
              },
              "input": {
                "type": "udp"
              },
              "agent": {
			  
			  
			 .....
			 
			 
			 "fortinet": {
                "firewall": {
                  "srcuuid": "39426",
                  "msg": "SSL user failed to logged in",
                  "reason": "sslvpn_login_unknown_user",
                  "remote": "xxx.xxx.xxx.xxxx",

GET _ingest/geoip/stats Returns ok without failures

{
  "stats": {
    "successful_downloads": 3,
    "failed_downloads": 0,
    "total_download_time": 11874,
    "databases_count": 3,
    "skipped_updates": 0,
    "expired_databases": 0
  },
  "nodes": {
    "k586sYdpRte5O_sDCZsEQg": {
      "databases": [
        {
          "name": "GeoLite2-ASN.mmdb"
        },
        {
          "name": "GeoLite2-City.mmdb"
        },
        {
          "name": "GeoLite2-Country.mmdb"
        }
      ],
      "files_in_temp": [
        "GeoLite2-Country.mmdb_LICENSE.txt",
        "GeoLite2-ASN.mmdb",
        "GeoLite2-ASN.mmdb_LICENSE.txt",
        "GeoLite2-City.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb_README.txt",
        "GeoLite2-Country.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb",
        "GeoLite2-Country.mmdb",
        "GeoLite2-City.mmdb_COPYRIGHT.txt"
      ],
      "cache_stats": {
        "count": 0,
        "hits": 0,
        "misses": 0,
        "evictions": 0,
        "hits_time_in_millis": 0,
        "misses_time_in_millis": 0
      }
    }
  }
}

can you help me?

Hi @Fernando_Bertoldi, Welcome to the community...

First, I am curious why you are not using the built-in Fortinet module; then all this would be taken care of automatically for you... do you have a specific reason why you are not using the module?

With the way you are doing it

  1. We would need to see what a document in json that is received in Elasticsearch and you should try to simulate.... we can not help debug unless you provide a sample.

  2. I would add a a set processor in your geoip pipeline and add a field like below to make sure the pipeline is actually called

{
    "set": {
      "field": "custom_pipeline",
      "value": "fortinet-geoip-pipeline"
    }
  },

3rd Your pipeline is writing to the same field geo that is not a good approach

"target_field": "geo",

should be like

          "target_field": "geo_remote",

....
          "target_field": "geo_local",

But in the end... why are you not using the module?

1 Like

Hello, thanks for the help.

To confirm the fortinet configuration, would it look like this?

config log syslogd setting
	set status enabled
	set server "x.x.x.x"
	set mode udp
	set port 9004
	set facility user
	set format default

end

The fortinet.module.yml should look like this?

- module: fortinet
  firewall:
    enabled: true
    var.input: udp
    var.syslog_host: 0.0.0.0
    var.syslog_port: 9004
    var.tags: ["fortinet-fortigate2", "fortinet-firewall2"]

The filebeat.yml

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

Do I need to add something else, like processors, pipelines, dissecs in fortinet.module or filebeat.yml
to see the geoip?

Regards

Looks good to me me Did you try it?

You might need to clean up the whole data because it may have created bad mappings /. index.

You need to run setup first BEFORE starting to load data.

I would refer to the filebeat quick start guide. Make sure you follow the steps.. And in your case, you're using Forinet not nginx

Hi, the fortinet module works wonderfully, thanks for the light.

1 Like