# Output logs to host based on timestamp

**URL:** <https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357>\
**Category:** Logstash\
**Created:** [August 25, 2020, 9:27pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357 "2020-08-25T21:27:33Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ManjaroBlack](https://avatars.discourse-cdn.com/v4/letter/m/ba9def/32.png) [@ManjaroBlack](https://discuss.elastic.co/u/ManjaroBlack)\
**Post date:** [August 25, 2020, 9:27pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/1 "2020-08-25T21:27:33Z")

</div>

I have logs that are a mix of old and new. I'd like to send any logs that are newer than 2 weeks to fast SSD, otherwise send to slower HDD. Can someone point me in the direction to figure this out?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 25, 2020, 9:28pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/2 "2020-08-25T21:28:48Z")

</div>

Welcome to our community! 😃

What output are you using?

---

<div class="post-metadata">

**Author:** ![ManjaroBlack](https://avatars.discourse-cdn.com/v4/letter/m/ba9def/32.png) [@ManjaroBlack](https://discuss.elastic.co/u/ManjaroBlack)\
**Post date:** [August 25, 2020, 9:30pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/3 "2020-08-25T21:30:57Z")

</div>

Thanks! I am outputting to elasticsearch on localhost:

```auto
    elasticsearch {
          index => "maximo-%{+YYYY.MM.dd}"
          hosts => "localhost:9200"
          document_id => "%{[@metadata][fingerprint]}"
    }

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 25, 2020, 10:34pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/4 "2020-08-25T22:34:06Z")

</div>

Ok, in that case this is not a Logstash thing, it's where Elasticsearch will be storing the indices.

You can use [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-start.html) to manage that.

---

<div class="post-metadata">

**Author:** ![ManjaroBlack](https://avatars.discourse-cdn.com/v4/letter/m/ba9def/32.png) [@ManjaroBlack](https://discuss.elastic.co/u/ManjaroBlack)\
**Post date:** [August 26, 2020, 12:45pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/5 "2020-08-26T12:45:44Z")

</div>

Thanks for that. I looked into ILM and I am not sure how it can solve my problem. Does ILM provide a way to identify which disk to place the index on based on the timestamps of the entries or the name of the index?

The idea I was thinking of, but I am not sure if this is how it works, if I output to different elasticsearch nodes, so running a master on localhost:9200 with fast storage, and another instance on localhost:9201 with slow storage and using an if statement in the output to direct logs to the right place based on the timestamp of the entry. Would something like this work? I am just not sure how to stup the if statement to make a comparision of the timestamp of the log entry to the current time.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 26, 2020, 9:34pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/6 "2020-08-26T21:34:43Z")

</div>

With ILM you can define "hot" data to go onto certain nodes, then as it ages it goes to "warm" and then moves to over nodes.

[https://www.elastic.co/guide/en/elasticsearch/reference/7.9/ilm-allocate.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/ilm-allocate.html) is where it goes into details on that.

---

<div class="post-metadata">

**Author:** ![ManjaroBlack](https://avatars.discourse-cdn.com/v4/letter/m/ba9def/32.png) [@ManjaroBlack](https://discuss.elastic.co/u/ManjaroBlack)\
**Post date:** [August 31, 2020, 4:24pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/7 "2020-08-31T16:24:35Z")

</div>

What about data that is already old, does it just have to wait for the specified time before it goes to the warm storage? Isn't there a way to store the old data that is getting ingested the for the first time straight on to the warm storage?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 31, 2020, 9:30pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/8 "2020-08-31T21:30:52Z")

</div>

You can set indices to be created on specific nodes using allocation filtering - [https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 28, 2020, 9:31pm UTC](https://discuss.elastic.co/t/output-logs-to-host-based-on-timestamp/246357/9 "2020-09-28T21:31:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
