# The correct use of logstash

**URL:** <https://discuss.elastic.co/t/the-correct-use-of-logstash/239823>\
**Category:** Logstash\
**Created:** [July 3, 2020, 2:42pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823 "2020-07-03T14:42:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Katia](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@Katia](https://discuss.elastic.co/u/Katia)\
**Post date:** [July 3, 2020, 2:42pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/1 "2020-07-03T14:42:09Z")

</div>

I have four types of log files.  
Every day I get the 4 types from several machines and, I have for each machine : t1-20200703, t2-20200703, t3-20200703 and t4-20200703.  
After a few days I will have hundreds of files and I want to know how I can organize all this on elasticsearch? and on kibana?  
I'm using logstash.

```auto
output{
	elasticsearch{
		hosts => ["localhost:9200"]
		index => "%{type}%{+yyyy.MM.dd}"
	}
	
}

```

Thank you

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 4, 2020, 3:05am UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/2 "2020-07-04T03:05:13Z")

</div>

There's not really enough info here to be helpful, but some general guidelines:

- You need to avoid too many indices / shards. You should probably combine log files if they are for the same application and almost certainly combine logs from different hosts.
- You should research ILM (index lifecycle management), it is the "new way" instead of date based indexes and automate management and deletion of old indices.

---

<div class="post-metadata">

**Author:** ![Katia](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@Katia](https://discuss.elastic.co/u/Katia)\
**Post date:** [July 6, 2020, 7:33am UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/3 "2020-07-06T07:33:14Z")

</div>

could you please explain how can I combine log files if they are for the same application ?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 6, 2020, 12:46pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/4 "2020-07-06T12:46:16Z")

</div>

Just send to the same index.

---

<div class="post-metadata">

**Author:** ![Katia](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@Katia](https://discuss.elastic.co/u/Katia)\
**Post date:** [July 6, 2020, 1:02pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/5 "2020-07-06T13:02:19Z")

</div>

my index takes the date as its name. so it automatically changes its name when there is new data !

---

<div class="post-metadata">

**Author:** ![chandra2037](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandra2037/32/72142_2.png) [@chandra2037](https://discuss.elastic.co/u/chandra2037)\
**Post date:** [July 7, 2020, 12:59pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/6 "2020-07-07T12:59:03Z")

</div>

As @rugenl pointed out using [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) will be the better way of automating the process of index management on the Elastic side.

If you think logs from different machines are of similar structure, then you can store logs from different machines into same index. Probably add host information to the logs, so that you can filter on these fields to get logs from different machines. Checkout [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) to standardize naming conventions.

As you start using ILM, no need to specify the particular index name, you will be providing the alias name which will be a static name. Elasticsearch will automatically indexes into a date based index and will rollover automatically based on your ILM policy.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2020, 12:59pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823/7 "2020-08-04T12:59:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.