# Dynamic Index Name Creation Based on Updated @Timestamp Field

**URL:** <https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 26, 2018, 9:43pm UTC](https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178 "2018-12-26T21:43:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![devops\_mike](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@devops\_mike](https://discuss.elastic.co/u/devops_mike)\
**Post date:** [December 26, 2018, 9:43pm UTC](https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178/1 "2018-12-26T21:43:17Z")

</div>

We would like to process old IIS Log files and have the data stored in indices that correspond to the event's date, and not the date which it was harvested. We are using version 6.5.3 for Elasticsearch, Filebeats, and the rest of our stack. IIS Module is enabled within Filebeats.yml.

We recently updated to use Monthly Indices, and this is working great. Based on some research and preliminary testing we are working on the following assumptions. If we're off on any of this we'd appreciate knowing.

The default.json file within the module\iis\access\ingest folder is where we update our grok pattern. Also within this file we see how the included @Timestamp field gets reset to the iis.access.time field from the event. The old @Timestamp field value is stored in read\_timestamp.

Within filebeat.yml, we setup our index to create monthly indices.

```auto
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["#{Elasticsearch.Dns.Name}:9200"]
  index: "filebeat-monthly-%{[beat.version]}-%{+yyyy.MM}"

```

However, when we tried to process IIS Logs from June, it stuffed them into `filebeat-monthly-6.5.3-2018.12`. Based on the Format String (sprintf) [documentation](https://www.elastic.co/guide/en/beats/libbeat/6.5/config-file-format-type.html#_format_string_sprintf) we expected the date for the index to be based on the `@Timestamp` field. Looking at the documents in the index the `@Timestamp` field is accurate, as is the `read_timestamp`. Those fields show the event was from June, but processed today.

Our assumption is `@Timestamp` should drive which index receives the documents. We must be mistaken because documents are going into the index based on the `read_timestamp` not the event `@Timestamp`. What are we missing?

Also, at what point does it determine the index name? We are curious if it got the @Timestamp field when it was still the read time. Would that mean the index name is decided before the ingest pipeline processors have a chance to update fields?

Thank you!

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 27, 2018, 2:50pm UTC](https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178/2 "2018-12-27T14:50:02Z")

</div>

The IIS modules uses Elastic Ingest Node for parsing. The log message it's timestamp is unknown to filebeat.

In Ingest Node one can change the index by overwriting the `_index` field in the event.

You will have to add another processor overwriting `_index` to your likings (e.g. via painless processor) and reinstall the pipeline.

---

<div class="post-metadata">

**Author:** ![devops\_mike](https://avatars.discourse-cdn.com/v4/letter/d/50afbb/32.png) [@devops\_mike](https://discuss.elastic.co/u/devops_mike)\
**Post date:** [January 2, 2019, 10:00pm UTC](https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178/3 "2019-01-02T22:00:38Z")

</div>

Hi Steffens I have been trying to get the pipeline working with the \_index rename, but I have been very unsuccessful.

My changes to "\filebeat\module\iis\access\ingest\default.json" do not appear to have the desired effects. It seemed rather simple, I tried both the script and the set processors to change the \_index field. But it didn't seem to matter what I set \_index to, it wouldn't use it. I was also changing the index pattern in filebeat.yml to use the \_index field.

To troubleshoot changes in default.json, I then tried using the SET processor to alter other fields. I saw no cause/effect for changes made in there. I tried many different fields, such as beat.name, iis.access.user\_agent.original, etc.. I tried accessing these fields with and without the \_source prefix.

In between changes, I stop and then start the filebeat service.

Example:

```auto
{
    "set":{
      "field": "beat.name",
      "value": "test"
    }

```

[https://www.elastic.co/guide/en/elasticsearch/reference/master/set-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/set-processor.html)

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 3, 2019, 2:45pm UTC](https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178/4 "2019-01-03T14:45:29Z")

</div>

When changing the file you have to force-update the pipeline (or remove the old pipeline from Elasticsearch).

E.g. run `filebeat setup --pipelines` to update ingest node pipelines.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 31, 2019, 2:45pm UTC](https://discuss.elastic.co/t/dynamic-index-name-creation-based-on-updated-timestamp-field/162178/5 "2019-01-31T14:45:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.