Filebeat output to custom index name not working when a template is used

Hello all,

The goal is to get data to go from filebeat > elasticsearch from custom app JSON logs and then add client.geo coordinates into the index for dashboard maps.

Originally, filebeat was sending all syslog and myapp JSON data to the default index filebeat-x.y.z and was working as expected.

Then, a pipeline (geoip-info) for adding client geo points was also added (specifically from here), everything appeared to be working.

[
  {
    "geoip": {
      "field": "client.ip",
      "target_field": "client.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "client.ip",
      "target_field": "client.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "src_ip",
      "target_field": "client.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "src_ip",
      "target_field": "client.as",
      "database_file": "GeoLite2-ASN.mmdb",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "source.ip",
      "target_field": "source.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "source.ip",
      "target_field": "source.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "destination.ip",
      "target_field": "destination.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "destination.ip",
      "target_field": "destination.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "server.ip",
      "target_field": "server.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "server.ip",
      "target_field": "server.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "host.ip",
      "target_field": "host.geo",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "server.as.asn",
      "target_field": "server.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "server.as.organization_name",
      "target_field": "server.as.organization.name",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "client.as.asn",
      "target_field": "client.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "client.as.organization_name",
      "target_field": "client.as.organization.name",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "source.as.asn",
      "target_field": "source.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "source.as.organization_name",
      "target_field": "source.as.organization.name",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "destination.as.asn",
      "target_field": "destination.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "destination.as.organization_name",
      "target_field": "destination.as.organization.name",
      "ignore_missing": true
    }
  }
]

Everything a-ok up to this point. Enter the issue...

Now it has been decided to separate the logs from "myapp" (in JSON format) to a different index from the index with all the other metrics from other systems in it.

Sadly, I am having no luck doing this...

To achieve this, I tried separating the indexes based on file input using the following filebeat.yml:

filebeat.inputs:
- type: filestream
  id: myapp-system
  paths:
    - /var/log/*.log
    - /var/log/syslog
  tags: ["syslog","myapp-system-logs"]
- type: log
  paths:
    - /home/myapp/myapp/var/log/myapp/myapp.json
  json:
    keys_under_root: true
    add_error_key: true
    expand_keys: true
    overwrite_keys: true
  enabled: true
  id: myapp-json-logs
  tags: ["json","myapp-logs"]
  index: "myapp-%{+yyyy.MM.dd}"
  pipeline: geoip-info

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ['https://<host:port>']
  ssl:
    enabled: true
    ca_trusted_fingerprint: "<fingerprint>"
  preset: balanced
  protocol: "https"
  username: "<username>"
  password: "<password"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

This appears to be ok:

user@system:~/ filebeat test config
Config OK

Followed by:

user@system:~/ systemctl restart filebeat

This appeared to work, I did indeed see the index myapp-2024.04.01 appear in Kibana > Index Management > Indices.

The success was short lived, migrating the dashboards in Kibana to the new index revealed that the new index (myapp) did not contain a "geo_point" field from the above pipline (although it should be noted that the pipeline's other fields like client.geo.city_name do work (indicating the pipeline does work (mostly?)).

Here is probably where I fail to understand how things are supposed to work.

I decided that this problem should be solved by adding an index template to add this information. I copied the mapping from the myapp index that filebeat created, then created a new template and simply added the geo-coordinates mapping (trying to mimic what I saw in the default filebeat template that does work).

The Index template looks like the below using the Index pattern myapp-*

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "myapp"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "dynamic_templates": [],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "arch": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "client": {
          "properties": {
            "as": {
              "properties": {
                "number": {
                  "type": "long"
                },
                "organization": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                }
              }
            },
            "geo": {
              "properties": {
                "city_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "continent_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "location": {
                  "properties": {
                    "lat": {
                      "type": "float"
                    },
                    "lon": {
                      "type": "float"
                    }
                  }
                },
                "region_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "availability_zone": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "image": {
              "properties": {
                "id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "provider": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "service": {
              "properties": {
                "name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "compCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "dst_ip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "dst_port": {
          "type": "long"
        },
        "duplicate": {
          "type": "boolean"
        },
        "duration": {
          "type": "float"
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "encCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "eventid": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "geo": {
          "properties": {
            "city_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "continent_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "continent_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "country_iso_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "country_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "location": {
              "type": "geo_point"
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "postal_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region_iso_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "timezone": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "hassh": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "hasshAlgorithms": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "height": {
          "type": "long"
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "containerized": {
              "type": "boolean"
            },
            "geo": {
              "properties": {
                "city_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "continent_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "location": {
                  "properties": {
                    "lat": {
                      "type": "float"
                    },
                    "lon": {
                      "type": "float"
                    }
                  }
                },
                "region_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "hostname": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "mac": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "os": {
              "properties": {
                "codename": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "family": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "kernel": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "platform": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "version": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "kexAlgs": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "keyAlgs": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "langCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "macCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "password": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "protocol": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "sensor": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "session": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "shasum": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "size": {
          "type": "long"
        },
        "src_ip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "src_port": {
          "type": "long"
        },
        "tags": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "timestamp": {
          "type": "date"
        },
        "ttylog": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "username": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "value": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "width": {
          "type": "long"
        }
      }
    },
    "aliases": {}
  }
}

I then delete the existing Index myapp-x.y.z expecting filebeat to create a new one using the template above defined in Kinaba. However this does not happen, in fact nothing at all happens...

I then change the template Index pattern to purposely not match the expected index name and then a minute later the expected Index myapp-x.y.z appears and works.

So it would seem that somehow my template is preventing creation of the index, but I am at a loss as to why. There is no indication in filebeat or elasticsearch logs that there is any issue and I have read every link I can find on Google and in the documentation (literally spent days) and I am defeated...

If there is any advice or perhaps even a smarter way to go about what I am trying to do, it would be super appreciated.

Ok, after more learning how to debug and losing hair I got to the bottom of it.

For anyone else who runs into the issue where your index appears to not be created when using a template, I followed these steps to fault find:

  1. In filebeat.yml, I enabled debug logging:
logging.level: debug
logging.to_files: true
logging.files:
 path: /var/log/filebeat
 name: filebeat
 keepfiles: 7
 permissions: 0640
  1. Restart filebeat:
user@system:~/ systemctl restart filebeat
  1. Inspected the latest log and found and event that was supposed to be sent to Elasticsearch, it was something like this:
{"log.level":"debug","@timestamp":"2024-04-01T16:04:17.120+0800","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2024-04-01T08:04:17.120Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.12.2\",\n    \"pipeline\": \"geoip-info\",\n    \"raw_index\": \"myapp-2024.04.01\"\n  },\n  \"src_ip\": \"1.1.1.1\",\n  \"eventid\": \"myapp.session.closed\",\n  \"duration\": 2.6971168518066406,\n  \"fields\": {\n    \"test\": true\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  },\n  \"sensor\": \"project-myapp-01\",\n  \"tags\": [\n    \"json\",\n    \"myapp-logs\",\n    \"my-app\"\n  ],\n  \"cloud\": {\n    \"image\": {\n      \"id\": \"ami-zxcf\"\n    },\n    \"provider\": \"aws\",\n    \"service\": {\n      \"name\": \"EC2\"\n    },\n    \"instance\": {\n      \"id\": \"i-fvvbbgf\"\n    },\n    \"machine\": {\n      \"type\": \"t2.micro\"\n    },\n    \"region\": \"<zone>\",\n    \"availability_zone\": \"<zone>\",\n    \"account\": {\n      \"id\": \"<id>\"\n    }\n  },\n  \"input\": {\n    \"type\": \"log\"\n  },\n  \"agent\": {\n    \"type\": \"filebeat\",\n    \"version\": \"8.12.2\",\n    \"ephemeral_id\": \"xxxx\",\n    \"id\": \"xxx\",\n    \"name\": \"myapp-01.test.com\"\n  },\n  \"host\": {\n    \"id\": \"xxxxx\",\n    \"containerized\": false,\n    \"ip\": [\n      \"192.168.1.5\",\n      \"::1\",\n      \"::1\"\n    ],\n    \"name\": \"myapp.test.com\",\n    \"mac\": [\n      \"02-D5-40-CC-4E-2F\"\n    ],\n    \"hostname\": \"myapp.test.com\",\n    \"architecture\": \"x86_64\",\n    \"os\": {\n      \"version\": \"12 (bookworm)\",\n      \"family\": \"debian\",\n      \"name\": \"Debian GNU/Linux\",\n      \"kernel\": \"6.1.0-18-cloud-amd64\",\n      \"codename\": \"bookworm\",\n      \"type\": \"linux\",\n      \"platform\": \"debian\"\n    }\n  },\n  \"session\": \"b1c77e3ccff5\",\n  \"message\": \"Connection lost after 2 seconds\",\n  \"timestamp\": \"2024-04-01T08:04:16.381394+0000\",\n  \"log\": {\n    \"file\": {\n      \"path\": \"/home/myapp/myapp/var/log/myapp/myapp.json\"\n    },\n    \"offset\": 247762\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
  1. I copied from @timestamp onwards. Went to a web site to remove all the escapes, I think this was it.

  2. I went to Stack Management > Dev tools and tried to manually paste it in:

POST myapp-2024.04.01/_doc/
{
  "@timestamp": "2024-04-01T07:43:57.985Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.2",
    "raw_index": "myapp-2024.04.01",
    "pipeline": "geoip-info"
  },

(I removed the rest from the above test, make sure you add all the JSON from the filebeat output in steps 4-5.

  1. I got a return telling me that "auto creation is not allowed in the template (or something to that effect).
  2. In my template, I turned on "Auto allow create".
  3. Did the post again in the dev tools and got:
{
  "_index": ".ds-myapp-2024.04.01-2024.04.01-000001",
  "_id": "xxxx",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 10,
  "_primary_term": 1
}

Now all works as expected.

If my assumptions are correct, filebeat should have seen the same error I did when it tried to POST the data like I did. It really seems like receiving an error from elasticsearch when trying to create an index is the type of thing that should show up in the filebeat logs. There was no indication of any issue even at debug level.