# Filebeat output to custom index name not working when a template is used

**URL:** <https://discuss.elastic.co/t/filebeat-output-to-custom-index-name-not-working-when-a-template-is-used/356510>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 1, 2024, 2:33am UTC](https://discuss.elastic.co/t/filebeat-output-to-custom-index-name-not-working-when-a-template-is-used/356510 "2024-04-01T02:33:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![DazzleNovak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dazzlenovak/32/111249_2.png) [@DazzleNovak](https://discuss.elastic.co/u/DazzleNovak)\
**Post date:** [April 1, 2024, 2:33am UTC](https://discuss.elastic.co/t/filebeat-output-to-custom-index-name-not-working-when-a-template-is-used/356510/1 "2024-04-01T02:33:38Z")

</div>

Hello all,

The goal is to get data to go from **filebeat \> elasticsearch** from custom app JSON logs and then add **client.geo coordinates** into the index for dashboard maps.

Originally, filebeat was sending all syslog and myapp JSON data to the default index **filebeat-x.y.z** and was working as expected.

Then, a pipeline ( **geoip-info** ) for adding client geo points was also added ([specifically from here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-geoip.html)), everything appeared to be working.

```auto
[
  {
    "geoip": {
      "field": "client.ip",
      "target_field": "client.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "client.ip",
      "target_field": "client.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "src_ip",
      "target_field": "client.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "src_ip",
      "target_field": "client.as",
      "database_file": "GeoLite2-ASN.mmdb",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "source.ip",
      "target_field": "source.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "source.ip",
      "target_field": "source.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "destination.ip",
      "target_field": "destination.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "destination.ip",
      "target_field": "destination.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "server.ip",
      "target_field": "server.geo",
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "database_file": "GeoLite2-ASN.mmdb",
      "field": "server.ip",
      "target_field": "server.as",
      "properties": [
        "asn",
        "organization_name"
      ],
      "ignore_missing": true
    }
  },
  {
    "geoip": {
      "field": "host.ip",
      "target_field": "host.geo",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "server.as.asn",
      "target_field": "server.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "server.as.organization_name",
      "target_field": "server.as.organization.name",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "client.as.asn",
      "target_field": "client.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "client.as.organization_name",
      "target_field": "client.as.organization.name",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "source.as.asn",
      "target_field": "source.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "source.as.organization_name",
      "target_field": "source.as.organization.name",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "destination.as.asn",
      "target_field": "destination.as.number",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "destination.as.organization_name",
      "target_field": "destination.as.organization.name",
      "ignore_missing": true
    }
  }
]

```

Everything a-ok up to this point. Enter the issue...

Now it has been decided to separate the logs from " **myapp**" (in JSON format) to a different index from the index with all the other metrics from other systems in it.

Sadly, I am having no luck doing this...

To achieve this, I tried separating the indexes based on file input using the following **filebeat.yml** :

```auto
filebeat.inputs:
- type: filestream
  id: myapp-system
  paths:
    - /var/log/*.log
    - /var/log/syslog
  tags: ["syslog","myapp-system-logs"]
- type: log
  paths:
    - /home/myapp/myapp/var/log/myapp/myapp.json
  json:
    keys_under_root: true
    add_error_key: true
    expand_keys: true
    overwrite_keys: true
  enabled: true
  id: myapp-json-logs
  tags: ["json","myapp-logs"]
  index: "myapp-%{+yyyy.MM.dd}"
  pipeline: geoip-info

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ['https://<host:port>']
  ssl:
    enabled: true
    ca_trusted_fingerprint: "<fingerprint>"
  preset: balanced
  protocol: "https"
  username: "<username>"
  password: "<password"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

This appears to be ok:

```bash
user@system:~/ filebeat test config
Config OK

```

Followed by:

```bash
user@system:~/ systemctl restart filebeat

```

This appeared to work, I did indeed see the index **myapp-2024.04.01** appear in **Kibana \> Index Management \> Indices**.

The success was short lived, migrating the dashboards in Kibana to the new index revealed that the new index (myapp) did not contain a " **geo\_point**" field from the above pipline (although it should be noted that the pipeline's other fields like **client.geo.city\_name** do work (indicating the pipeline does work (mostly?)).

Here is probably where I fail to understand how things are supposed to work.

I decided that this problem should be solved by adding an index template to add this information. I copied the mapping from the **myapp** index that filebeat created, then created a new template and simply added the geo-coordinates mapping (trying to mimic what I saw in the default **filebeat** template that does work).

The Index template looks like the below using the Index pattern **myapp-** \*

```auto
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "myapp"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "dynamic_templates": [],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "arch": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "client": {
          "properties": {
            "as": {
              "properties": {
                "number": {
                  "type": "long"
                },
                "organization": {
                  "properties": {
                    "name": {
                      "type": "text",
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      }
                    }
                  }
                }
              }
            },
            "geo": {
              "properties": {
                "city_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "continent_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "location": {
                  "properties": {
                    "lat": {
                      "type": "float"
                    },
                    "lon": {
                      "type": "float"
                    }
                  }
                },
                "region_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "availability_zone": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "image": {
              "properties": {
                "id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "provider": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "service": {
              "properties": {
                "name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "compCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "dst_ip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "dst_port": {
          "type": "long"
        },
        "duplicate": {
          "type": "boolean"
        },
        "duration": {
          "type": "float"
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "encCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "eventid": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "geo": {
          "properties": {
            "city_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "continent_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "continent_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "country_iso_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "country_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "location": {
              "type": "geo_point"
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "postal_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region_iso_code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region_name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "timezone": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "hassh": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "hasshAlgorithms": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "height": {
          "type": "long"
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "containerized": {
              "type": "boolean"
            },
            "geo": {
              "properties": {
                "city_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "continent_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "country_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "location": {
                  "properties": {
                    "lat": {
                      "type": "float"
                    },
                    "lon": {
                      "type": "float"
                    }
                  }
                },
                "region_iso_code": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "region_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "hostname": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "mac": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "os": {
              "properties": {
                "codename": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "family": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "kernel": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "platform": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "type": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "version": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "kexAlgs": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "keyAlgs": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "langCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "macCS": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "password": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "protocol": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "sensor": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "session": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "shasum": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "size": {
          "type": "long"
        },
        "src_ip": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "src_port": {
          "type": "long"
        },
        "tags": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "timestamp": {
          "type": "date"
        },
        "ttylog": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "username": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "value": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "width": {
          "type": "long"
        }
      }
    },
    "aliases": {}
  }
}

```

I then delete the existing Index myapp-x.y.z expecting filebeat to create a new one using the template above defined in Kinaba. However this does not happen, in fact nothing at all happens...

I then change the template Index pattern to purposely not match the expected index name and then a minute later the expected Index myapp-x.y.z appears and works.

So it would seem that somehow my template is preventing creation of the index, but I am at a loss as to why. There is no indication in filebeat or elasticsearch logs that there is any issue and I have read every link I can find on Google and in the documentation (literally spent days) and I am defeated...

If there is any advice or perhaps even a smarter way to go about what I am trying to do, it would be super appreciated.

---

<div class="post-metadata">

**Author:** ![DazzleNovak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dazzlenovak/32/111249_2.png) [@DazzleNovak](https://discuss.elastic.co/u/DazzleNovak)\
**Post date:** [April 1, 2024, 8:31am UTC](https://discuss.elastic.co/t/filebeat-output-to-custom-index-name-not-working-when-a-template-is-used/356510/2 "2024-04-01T08:31:23Z")

</div>

Ok, after more learning how to debug and losing hair I got to the bottom of it.

For anyone else who runs into the issue where your index appears to not be created when using a template, I followed these steps to fault find:

1. In filebeat.yml, I enabled debug logging:

```auto
logging.level: debug
logging.to_files: true
logging.files:
 path: /var/log/filebeat
 name: filebeat
 keepfiles: 7
 permissions: 0640

```

1. Restart filebeat:

```bash
user@system:~/ systemctl restart filebeat

```

1. Inspected the latest log and found and event that was supposed to be sent to Elasticsearch, it was something like this:

```auto
{"log.level":"debug","@timestamp":"2024-04-01T16:04:17.120+0800","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/processing.debugPrintProcessor.func1","file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n \"@timestamp\": \"2024-04-01T08:04:17.120Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.12.2\",\n \"pipeline\": \"geoip-info\",\n \"raw_index\": \"myapp-2024.04.01\"\n },\n \"src_ip\": \"1.1.1.1\",\n \"eventid\": \"myapp.session.closed\",\n \"duration\": 2.6971168518066406,\n \"fields\": {\n \"test\": true\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n },\n \"sensor\": \"project-myapp-01\",\n \"tags\": [\n \"json\",\n \"myapp-logs\",\n \"my-app\"\n],\n \"cloud\": {\n \"image\": {\n \"id\": \"ami-zxcf\"\n },\n \"provider\": \"aws\",\n \"service\": {\n \"name\": \"EC2\"\n },\n \"instance\": {\n \"id\": \"i-fvvbbgf\"\n },\n \"machine\": {\n \"type\": \"t2.micro\"\n },\n \"region\": \"<zone>\",\n \"availability_zone\": \"<zone>\",\n \"account\": {\n \"id\": \"<id>\"\n }\n },\n \"input\": {\n \"type\": \"log\"\n },\n \"agent\": {\n \"type\": \"filebeat\",\n \"version\": \"8.12.2\",\n \"ephemeral_id\": \"xxxx\",\n \"id\": \"xxx\",\n \"name\": \"myapp-01.test.com\"\n },\n \"host\": {\n \"id\": \"xxxxx\",\n \"containerized\": false,\n \"ip\": [\n \"192.168.1.5\",\n \"::1\",\n \"::1\"\n],\n \"name\": \"myapp.test.com\",\n \"mac\": [\n \"02-D5-40-CC-4E-2F\"\n],\n \"hostname\": \"myapp.test.com\",\n \"architecture\": \"x86_64\",\n \"os\": {\n \"version\": \"12 (bookworm)\",\n \"family\": \"debian\",\n \"name\": \"Debian GNU/Linux\",\n \"kernel\": \"6.1.0-18-cloud-amd64\",\n \"codename\": \"bookworm\",\n \"type\": \"linux\",\n \"platform\": \"debian\"\n }\n },\n \"session\": \"b1c77e3ccff5\",\n \"message\": \"Connection lost after 2 seconds\",\n \"timestamp\": \"2024-04-01T08:04:16.381394+0000\",\n \"log\": {\n \"file\": {\n \"path\": \"/home/myapp/myapp/var/log/myapp/myapp.json\"\n },\n \"offset\": 247762\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}

```

1. I copied from `@timestamp` onwards. Went to a web site to remove all the escapes, [I think this was it](https://www.freeformatter.com/json-escape.html).

2. I went to Stack Management \> Dev tools and tried to manually paste it in:

```auto
POST myapp-2024.04.01/_doc/
{
  "@timestamp": "2024-04-01T07:43:57.985Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.12.2",
    "raw_index": "myapp-2024.04.01",
    "pipeline": "geoip-info"
  },

```

(I removed the rest from the above test, make sure you add all the JSON from the filebeat output in steps 4-5.

1. I got a return telling me that "auto creation is not allowed in the template (or something to that effect).
2. In my template, I turned on " **Auto allow create**".
3. Did the post again in the dev tools and got:

```auto
{
  "_index": ".ds-myapp-2024.04.01-2024.04.01-000001",
  "_id": "xxxx",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 10,
  "_primary_term": 1
}

```

Now all works as expected.

If my assumptions are correct, filebeat should have seen the same error I did when it tried to POST the data like I did. It really seems like receiving an error from elasticsearch when trying to create an index is the type of thing that should show up in the filebeat logs. There was no indication of any issue even at debug level.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2024, 10:31am UTC](https://discuss.elastic.co/t/filebeat-output-to-custom-index-name-not-working-when-a-template-is-used/356510/3 "2024-04-29T10:31:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.