# How to ignore old logs

**URL:** <https://discuss.elastic.co/t/how-to-ignore-old-logs/83954>\
**Category:** Logstash\
**Created:** [April 28, 2017, 1:43am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954 "2017-04-28T01:43:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Raja1](https://avatars.discourse-cdn.com/v4/letter/r/85f322/32.png) [@Raja1](https://discuss.elastic.co/u/Raja1)\
**Post date:** [April 28, 2017, 1:43am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/1 "2017-04-28T01:43:59Z")

</div>

Hi All,

I have scenario like below.

Gatewaylogs --\> Logstash --\> Kafka --\> Logstash --\>Elasticsearch --\> Kibana

I am able to push the logs my machine to Kibana.  
Now the issue is whenever i am doing some changes in configuration. I am running every service to index the logs in Kibana. The same time old logs are again appending into new run.

How to ignore the logs from source server once it reaches to Kibana?

Regards  
Raja

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 28, 2017, 8:57am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/2 "2017-04-28T08:57:15Z")

</div>

Why are old logs being sent in the first place? What is this "Gatewaylogs" component?

---

<div class="post-metadata">

**Author:** ![Raja1](https://avatars.discourse-cdn.com/v4/letter/r/85f322/32.png) [@Raja1](https://discuss.elastic.co/u/Raja1)\
**Post date:** [April 28, 2017, 9:03am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/3 "2017-04-28T09:03:43Z")

</div>

Gatewaylogs are webservice logs where those are in server so i mentioned Gatewaylogs.

Old logs means-- When first time i ran the logstash pipeline the log1 will be pushed to Kibana. After adding one more log path in configuration and run one more time its taking first run log1 and second run log2.

So now i am seeing log1 is 2 times and log2 is one time.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 28, 2017, 9:10am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/4 "2017-04-28T09:10:44Z")

</div>

Yes, but why are the logs even read a second time? A standard configuration of Logstash won't read the same file twice. What does your configuration look like?

---

<div class="post-metadata">

**Author:** ![Raja1](https://avatars.discourse-cdn.com/v4/letter/r/85f322/32.png) [@Raja1](https://discuss.elastic.co/u/Raja1)\
**Post date:** [April 28, 2017, 9:13am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/5 "2017-04-28T09:13:20Z")

</div>

Here i am getting the logs from kafka.

input  
{  
kafka  
{  
bootstrap\_servers =\> "x.x.x.x:9092"  
topics =\>["logstash"]  
group\_id =\> "test-consumer-group"  
consumer\_threads =\> 1  
codec =\> "json"  
type =\> "%{type}"  
}  
}  
output  
{  
elasticsearch  
{  
hosts =\> ["x.x.x.x:9200"]  
}  
stdout {codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 28, 2017, 9:16am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/6 "2017-04-28T09:16:00Z")

</div>

I'm sure the kafka input records the current position so it doesn't process everything from the beginning each time Logstash is run, but I haven't used it myself so I can't really help out.

---

<div class="post-metadata">

**Author:** ![Raja1](https://avatars.discourse-cdn.com/v4/letter/r/85f322/32.png) [@Raja1](https://discuss.elastic.co/u/Raja1)\
**Post date:** [April 28, 2017, 9:27am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/7 "2017-04-28T09:27:40Z")

</div>

Okay Thank you Magnus.

AS i told you initially my architecture is not that way good but i have to make it by using that architecture.

I am shipping the logs by using logstash then kafka just processing the logs to again other logstash then finally it will be in kibana thru elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 26, 2017, 9:41am UTC](https://discuss.elastic.co/t/how-to-ignore-old-logs/83954/8 "2017-05-26T09:41:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.