# Index name from log date not from system time

**URL:** <https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782>\
**Category:** Logstash\
**Created:** [August 11, 2017, 1:28pm UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782 "2017-08-11T13:28:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![hasan\_sharekhan](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hasan\_sharekhan](https://discuss.elastic.co/u/hasan_sharekhan)\
**Post date:** [August 11, 2017, 1:28pm UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782/1 "2017-08-11T13:28:15Z")

</div>

Hi,

Today i.e on 11-Aug-2017 I am pushing back dated logs into elastic search via logstash via filebeats

I want to create index names of log dates not of current date.

Back dates logs are from Mar-Jul 2017, but for all these logs only 1 index is created by name "myindex-110817"

I want it to create separate index for each date given in log file, below is my configuration.

I want to use "mylogtime" field for creating index names

```
filter {
   grok {
      patterns_dir => ["config/tt_tradingserver/pattern"]	
      match => { "message" => "%{greedydata_end_nospace:mylogtime}%{zero_or_more_space}\[%{tt_ts_log_type:logtime}\]%{zero_or_more_space}\"%{zero_or_more_space}%{greedydata_end_nospace:logmsg}%{zero_or_more_space}\"" }
}

date {
   match => ["mylogtime" , "yyyy-mm-dd hh:mm:ss.ssszz"]
   target => "mylogtime"
   }
}

output {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "myindex-%{+ddmmyy}" 
  } }
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 11, 2017, 3:32pm UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782/2 "2017-08-11T15:32:24Z")

</div>

The date pattern in your date filter is wrong is various ways. Please consult the documentation on which tokens to use for various timestamp components.

Can't you parse the date into the default `@timestamp` field (i.e. delete the `target` option in your date filter)? Then your current configuration will almost work out of the box (`%{+ddmmyy}` needs a slight adjustment).

---

<div class="post-metadata">

**Author:** ![hasan\_sharekhan](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hasan\_sharekhan](https://discuss.elastic.co/u/hasan_sharekhan)\
**Post date:** [August 14, 2017, 9:56am UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782/3 "2017-08-14T09:56:00Z")

</div>

Thanks for your reply, tried as per your suggestion didnt work, if possible any small example please

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 14, 2017, 10:18am UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782/4 "2017-08-14T10:18:29Z")

</div>

Can you parse the date into the default `@timestamp` field instead of insisting on your `mylogtime` field?

---

<div class="post-metadata">

**Author:** ![hasan\_sharekhan](https://avatars.discourse-cdn.com/v4/letter/h/eada6e/32.png) [@hasan\_sharekhan](https://discuss.elastic.co/u/hasan_sharekhan)\
**Post date:** [August 17, 2017, 6:09am UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782/5 "2017-08-17T06:09:27Z")

</div>

Thanks, it is working as expected, changes that I made is in GROK filter instead of "t" i used "timestamp" and I changed by date filter as

```
date {
	match => ["timestamp" , "yyyy-MM-dd HH:mm:ss.SSSZZ"]
}

```

After the above changes I was getting 2 timestamp fields

1st =\> @timestamp July 28th 2017, 06:09:09.062  
2nd =\> timestamp 2017-07-28 06:09:09.062+05:30

Since i don't want to maintain 2 timestamp field, I used "remove\_field" for removing timestamp.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2017, 6:09am UTC](https://discuss.elastic.co/t/index-name-from-log-date-not-from-system-time/96782/6 "2017-09-14T06:09:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.