Index name when using an index template

Elastic Stack Elasticsearch
1

Hi, I want to set up an index template for data coming from Zenarmor (using Elasticsearch 8.14.3)

When the indices are created without an index template, the names are like this "zen2_conn-240726", where 240726 represents the current day. As soon as I have created an index template that matches zen2_conn2* the new index has the name zen2_conn_write.

In addition, the index template has these settings:

{
  "index": {
    "number_of_shards": "1",
    "number_of_replicas": "0"
  }
}

However, the new indes is created with 1 replica. Why is that? The index template is clearly applied, but the index setting is ignored?

Is there a way to define the name that's given to an index which matches a template?

Thank you!

2

Probably there are multiple index templates and some index templates override the settings.
You can use Simulate index API and see which index templates (both composable and legacy) affect for the new index creation.

GET _index_template/_simulate_index/zen2_conn-240726

Note: You can increase the priority of an index template. The highest priority will overlap the others settings in other index templates. Or simply remove others if not needed.

3

When chaning the index pattern to match "123", the result is empty:

{}

When the index pattern matches the name, the correct one is called, I set the number of replicas to 555 to have an indicator. So the template is being used

grafik
grafik1838×410 54.4 KB

Here's the whole template, is there maybe anything wrong with it?

{
  "template": {
    "settings": {
      "index": {
        "number_of_shards": "1",
        "number_of_replicas": "555",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "app_category": {
          "type": "keyword"
        },
        "app_id": {
          "type": "integer"
        },
        "app_name": {
          "type": "keyword"
        },
        "app_proto": {
          "type": "keyword"
        },
        "cloud_policyid": {
          "type": "keyword"
        },
        "community_id": {
          "type": "keyword"
        },
        "conn_uuid": {
          "type": "keyword"
        },
        "device": {
          "properties": {
            "category": {
              "type": "keyword"
            },
            "id": {
              "type": "keyword"
            },
            "name": {
              "type": "keyword"
            },
            "os": {
              "type": "keyword"
            },
            "osver": {
              "type": "keyword"
            },
            "vendor": {
              "type": "keyword"
            }
          }
        },
        "direction": {
          "type": "keyword"
        },
        "dst_dir": {
          "type": "keyword"
        },
        "dst_geoip": {
          "properties": {
            "area": {
              "type": "integer"
            },
            "asn": {
              "type": "integer"
            },
            "city_name": {
              "type": "keyword"
            },
            "continent_code": {
              "type": "keyword"
            },
            "country_code2": {
              "type": "keyword"
            },
            "country_code3": {
              "type": "keyword"
            },
            "country_name": {
              "type": "keyword"
            },
            "dma_code": {
              "type": "integer"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            },
            "metro": {
              "type": "integer"
            },
            "postal_code": {
              "type": "keyword"
            },
            "region_code": {
              "type": "keyword"
            },
            "region_name": {
              "type": "keyword"
            },
            "timezone": {
              "type": "keyword"
            }
          }
        },
        "dst_hostname": {
          "type": "keyword"
        },
        "dst_hwaddr": {
          "type": "keyword"
        },
        "dst_nbytes": {
          "type": "long"
        },
        "dst_npackets": {
          "type": "integer"
        },
        "dst_pbytes": {
          "type": "long"
        },
        "dst_tcp_flags": {
          "type": "keyword"
        },
        "dst_username": {
          "type": "keyword"
        },
        "encryption": {
          "type": "keyword"
        },
        "end_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "handshake_result": {
          "type": "keyword"
        },
        "input": {
          "type": "integer"
        },
        "interface": {
          "type": "keyword"
        },
        "ip_dst_port": {
          "type": "integer"
        },
        "ip_dst_saddr": {
          "type": "ip"
        },
        "ip_src_port": {
          "type": "integer"
        },
        "ip_src_saddr": {
          "type": "ip"
        },
        "is_blocked": {
          "type": "integer"
        },
        "is_local": {
          "type": "integer"
        },
        "output": {
          "type": "integer"
        },
        "policyid": {
          "type": "keyword"
        },
        "security_tags": {
          "type": "keyword"
        },
        "src_dir": {
          "type": "keyword"
        },
        "src_geoip": {
          "properties": {
            "area": {
              "type": "integer"
            },
            "asn": {
              "type": "integer"
            },
            "city_name": {
              "type": "keyword"
            },
            "continent_code": {
              "type": "keyword"
            },
            "country_code2": {
              "type": "keyword"
            },
            "country_code3": {
              "type": "keyword"
            },
            "country_name": {
              "type": "keyword"
            },
            "dma_code": {
              "type": "integer"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            },
            "metro": {
              "type": "integer"
            },
            "postal_code": {
              "type": "keyword"
            },
            "region_code": {
              "type": "keyword"
            },
            "region_name": {
              "type": "keyword"
            },
            "timezone": {
              "type": "keyword"
            }
          }
        },
        "src_hostname": {
          "type": "keyword"
        },
        "src_hwaddr": {
          "type": "keyword"
        },
        "src_nbytes": {
          "type": "long"
        },
        "src_npackets": {
          "type": "integer"
        },
        "src_pbytes": {
          "type": "long"
        },
        "src_tcp_flags": {
          "type": "keyword"
        },
        "src_username": {
          "type": "keyword"
        },
        "start_time": {
          "type": "date",
          "format": "epoch_millis"
        },
        "tags": {
          "type": "keyword"
        },
        "transport_proto": {
          "type": "keyword"
        },
        "vlanid": {
          "type": "keyword"
        },
        "web_actions": {
          "type": "keyword"
        },
        "web_actions_description": {
          "type": "keyword"
        }
      }
    },
    "aliases": {}
  }
}

Thank you!

4

You're welcome Andre, thanks for your feedback!
It looks correct to me. When you check the index pattern 123 it should return empty results if there is no index pattern that include every index like *.

The template you shared is a _component_template. There must be an _index_template too.

5

So the naming issue went away by itself (?). I decided to leave it at that with the index template activated. When I returned an hour later the zen2_conn_write index was gone and the index was created like before with the name in it. Haven't looked to find this behaviour documented somewhere yet. But I can confirm the template is being used with all settings correctly now.

Weird. Thanks a lot for your help.

6

Just tried to reproduce by deleting the index and have it created again.

In fact, the index is newly created using the _write suffix. It seems that at the full hour, the index was renamed to contain the date instead of write. So there seems to be some background magic going on that is configured somewhere else.

7

You're welcome Andre. Happy to hear it fixed.

1 Like