# Logstash not picking the file after placing in the input path

**URL:** <https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703>\
**Category:** Logstash\
**Created:** [May 12, 2019, 3:51pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703 "2019-05-12T15:51:35Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 14, 2019, 1:24pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/21 "2019-05-14T13:24:57Z")

</div>

Let me check that Badger that might be the issue though 🙂

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 14, 2019, 1:47pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/22 "2019-05-14T13:47:28Z")

</div>

Have changed that to just \_298 and then drop

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 14, 2019, 1:49pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/23 "2019-05-14T13:49:11Z")

</div>

But still no luck and it is not pciking it ☹

---

<div class="post-metadata">

**Author:** ![Jasonespo](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\
**Post date:** [May 14, 2019, 1:51pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/24 "2019-05-14T13:51:21Z")

</div>

You are restarting your service to pick up the config changes right?

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 14, 2019, 2:21pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/25 "2019-05-14T14:21:56Z")

</div>

Yes Jasonespo have done that still no luck

---

<div class="post-metadata">

**Author:** ![Jasonespo](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\
**Post date:** [May 14, 2019, 2:33pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/26 "2019-05-14T14:33:09Z")

</div>

Can you forward the update you made to:

> [@syed\_faheem\_Hussain](#):
>
> if ([message] !~ "Access\_298") {  
> drop { }  
> }

I believe Badger gave you the answer but I am not sure you implemented it correctly.

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 15, 2019, 1:32pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/27 "2019-05-15T13:32:03Z")

</div>

I have put filebeat now and it is working fine

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 15, 2019, 1:38pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/28 "2019-05-15T13:38:22Z")

</div>

May i know where i can post a new query ? I need to know how do we restrict the no of days data in elastic search

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2019, 2:19pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/29 "2019-05-15T14:19:22Z")

</div>

That's really an elasticsearch question more than a logstash question.

I would start by reading about [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html). If you are running an old version of the stack then you can do it using [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html).

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 15, 2019, 5:16pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/30 "2019-05-15T17:16:28Z")

</div>

Can i get a sample on how on how to put the delete API, i want to see the index logs in Kibana only for say last 3 days and delete others

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2019, 5:36pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/31 "2019-05-15T17:36:16Z")

</div>

I would ask that question in the elasticsearch forum.

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 15, 2019, 5:47pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/32 "2019-05-15T17:47:25Z")

</div>

> [@Badger](#):
>
> that ques

Hi Badger,

May i know the link for Elasticsearch forum please?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [May 15, 2019, 6:07pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/33 "2019-05-15T18:07:49Z")

</div>

[https://discuss.elastic.co/c/elasticsearch](https://discuss.elastic.co/c/elasticsearch)

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 15, 2019, 6:11pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/34 "2019-05-15T18:11:14Z")

</div>

Thank you have written a query there

---

<div class="post-metadata">

**Author:** ![Jasonespo](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\
**Post date:** [May 17, 2019, 9:48am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/35 "2019-05-17T09:48:19Z")

</div>

List all indices with:

GET \_cat/indices

```
green open index1 qi6sY0mEQFi5dFKx1WUGeA 1 1 0 0 7.4mb 179.6kb
green open index2 W65-VCneQ8CmjBhB-0JRxQ 5 1 61383 5 18.8mb 9.4mb

```

Delete an index with:

DELETE index1

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 17, 2019, 6:21pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/36 "2019-05-17T18:21:57Z")

</div>

I actually want to delete the data inside index prior to last two days and keep only last two days data how do i acheive it

---

<div class="post-metadata">

**Author:** ![Jasonespo](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)\
**Post date:** [May 17, 2019, 7:33pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/37 "2019-05-17T19:33:52Z")

</div>

Well.. you should make sure you index your data with a date timestamp on it.

Our production env uses index.date

So we can differentiate the indexes daily

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [May 17, 2019, 8:17pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/38 "2019-05-17T20:17:31Z")

</div>

delete\_by\_query can do that. but if you have too much data it will be expensive.  
for eaxmple

```
POST <index_name>/_delete_by_query
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "01-01-2019",
        "lte": "01-02-2019",
        "format": "MM-dd-yyyy"
       }
    }
  }
}

```

This can delete data which has timestamp two days. you just have to config first if your @timestamp field is same pattern MM-dd-yyyy

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 17, 2019, 8:46pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/39 "2019-05-17T20:46:07Z")

</div>

Are you rolling indexes over daily? Is a solution that deletes all indexes older than two days useful, or do you need to delete the subset of data in an index that is older than two days? The first is really cheap, the second is expensive.

---

<div class="post-metadata">

**Author:** ![syed\_faheem\_Hussain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/syed_faheem_hussain/32/46047_2.png) [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)\
**Post date:** [May 18, 2019, 11:41am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/40 "2019-05-18T11:41:09Z")

</div>

Hi Badger,

I want to delete the subset of data in the index which is two days old, also may I know in which file do we need to place this will it come in Logstash or Elasticsearch and what is the file name is it yml or properties file

[Previous page](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703.md?page=1)

[Next page](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703.md?page=3)