# Create a new index in elasticsearch for each log file by date

**URL:** <https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607>\
**Category:** Logstash\
**Created:** [October 21, 2016, 11:22am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607 "2016-10-21T11:22:47Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![adhikz](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@adhikz](https://discuss.elastic.co/u/adhikz)\
**Post date:** [October 21, 2016, 11:22am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/1 "2016-10-21T11:22:47Z")

</div>

Hi, I have managed to get my ELK stack up and ruining which filters logs using grok and passes data to elasticsearch  
and allows me to visualize in Kibana.

**Currently**  
I have completed the above task by using one log file and passes data with logstash to one index in elasticsearch :

`yellow open logstash-2016.10.19 5 1 1000807 0 364.8mb 364.8mb`

**What I actually want to do**

If i have the following logs files which are named according to Year,Month and Date

```
MyLog-2016-10-16.log
MyLog-2016-10-17.log
MyLog-2016-10-18.log
MyLog-2016-11-05.log
MyLog-2016-11-02.log
MyLog-2016-11-03.log

```

I would like to tell logstash to read by Year,Month and Date and create the following indexes :

`yellow open MyLog-2016-10-16.log 5 1 1000807 0 364.8mb 364.8mb`  
`yellow open MyLog-2016-10-17.log 1 1000807 0 364.8mb 364.8mb`  
`yellow open MyLog-2016-10-18.log 5 1 1000807 0 364.8mb 364.8mb`  
`yellow open MyLog-2016-11-05.log 5 1 1000807 0 364.8mb 364.8mb`  
`yellow open MyLog-2016-11-02.log 5 1 1000807 0 364.8mb 364.8mb`  
`yellow open MyLog-2016-11-03.log 5 1 1000807 0 364.8mb 364.8mb`

Please could I have some guidance as to how do i need to go about doing this ?

Thanks You

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 21, 2016, 11:27am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/2 "2016-10-21T11:27:30Z")

</div>

You are aware that indexes have a fixed memory overhead and that having too many indexes is a bad idea? With one index per logfile per day the numbers will quickly run up.

The index name is set with the `index` option of the elasticsearch output. It supports `%{fieldname}` references, so if you have a `logfilename` field containing the filename you could do this:

```nohighlight
output {
  elasticsearch {
    ...
    index => "%{logfilename}-%{+YYYY.MM.dd}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![adhikz](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@adhikz](https://discuss.elastic.co/u/adhikz)\
**Post date:** [October 21, 2016, 11:33am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/3 "2016-10-21T11:33:36Z")

</div>

> You are aware that indexes have a fixed memory overhead and that having too many indexes is a bad idea?

No I was not aware of this thanks for your input.

---

<div class="post-metadata">

**Author:** ![adhikz](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@adhikz](https://discuss.elastic.co/u/adhikz)\
**Post date:** [October 21, 2016, 11:38am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/4 "2016-10-21T11:38:15Z")

</div>

What about my filepath how do i get it to read each log file?

> input {  
> file {  
> **path** =\> "C:\Elk\logstash\bin\MyLog-2016-10-16.log"  
> start\_position =\> "beginning"  
> }  
> }

---

<div class="post-metadata">

**Author:** ![adhikz](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@adhikz](https://discuss.elastic.co/u/adhikz)\
**Post date:** [October 21, 2016, 11:47am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/5 "2016-10-21T11:47:30Z")

</div>

```
input {
	file {
	path => "C:\Elk\logstash\bin\%{MyLog}-%{+2016-10-16}"
	path => "C:\Elk\logstash\bin\%{MyLog}-%{+2016-10-17}"
	path => "C:\Elk\logstash\bin\%{MyLog}-%{+2016-10-18}"
	
	}
}

filter {
  grok {
    match => { "message" => "%?terms=%{WORD:key_word}*" }
  }
}

output {  
	elasticsearch { 
	
	index => %{MyLog}-%{+2016-10-16}"
	index => %{MyLog}-%{+2016-10-17}"
	index => %{MyLog}-%{+2016-10-18}"
	hosts => ["localhost:9200"] 

```

`}`

could i maybe do the above ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 21, 2016, 12:18pm UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/6 "2016-10-21T12:18:51Z")

</div>

> What about my filepath how do i get it to read each log file?

Use a wildcard like MyLog-\*.log. The actual path of the file from which an event was read can be found in the `path` field.

> could i maybe do the above ?

No, that doesn't make sense.

---

<div class="post-metadata">

**Author:** ![adhikz](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@adhikz](https://discuss.elastic.co/u/adhikz)\
**Post date:** [October 21, 2016, 12:21pm UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/7 "2016-10-21T12:21:50Z")

</div>

Thank very much for your reply. Im going to give it a try and will get back to you with feedback.

---

<div class="post-metadata">

**Author:** ![adhikz](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@adhikz](https://discuss.elastic.co/u/adhikz)\
**Post date:** [October 27, 2016, 1:38pm UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/8 "2016-10-27T13:38:32Z")

</div>

Hi Magnus,

I have created a new question thta i need help with. please could you take a look i will really appreciate it Link : _[Querying elastic search in Kibana using Json](https://discuss.elastic.co/t/querying-elastic-search-in-kibana-using-json/64150)_

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:32am UTC](https://discuss.elastic.co/t/create-a-new-index-in-elasticsearch-for-each-log-file-by-date/63607/9 "2017-07-06T04:32:20Z")

</div>