# What logstash input plugins can you set sincedb\_path for?

**URL:** <https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443>\
**Category:** Logstash\
**Created:** [June 25, 2019, 11:44pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443 "2019-06-25T23:44:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [June 25, 2019, 11:44pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443/1 "2019-06-25T23:44:17Z")

</div>

I have seen some Logstash configuration online like this:

```
input {
  file {
    start_position => "beginning"
    sincedb_path => "/pathToSincedb/sincedb"
    path => "/pathToYourData/*"
  }
}

```

As we can see, there is the `sincedb_path` that is set for the File input plugin.

**Question:**

1. What if the input plugin is `http_poller` or `cloudwatch_logs`, can I still use sincedb\_path?
2. If yes, what should I set my sincedb\_path as if I am going to use a docker? With dockers, things are a bit different from usual path setup. I've heard I need to mount the path in my docker file.
3. If no, how can I tell my program where it left off when reading the logs again?

**Objective:**

- I understand that sincedb\_path is by default set to null, but I don't want that. I want my program to keep track of what logs have been read so far, so the next time I restart my program again, it can start from where it left off.
- Setup sincedb\_path in my Logstash configuration that is used in a docker file.

I appreciate any feedback/suggestions.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 26, 2019, 1:44am UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443/2 "2019-06-26T01:44:32Z")

</div>

> [@EZprogramming](#):
>
> I understand that sincedb\_path is by default set to null, but I don't want that. I want my program to keep track of what logs have been read so far, so the next time I restart my program again, it can start from where it left off.

No, by default the file input will generate a valid (but obscure) sincedb\_path and persist the in-memory sincedb across restarts. Only a file input uses a sincedb to persist state. A jdbc input can use sql\_last\_value. It is possible that other inputs use other options, but I have no experience of those.

---

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [June 26, 2019, 8:58pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443/3 "2019-06-26T20:58:03Z")

</div>

@Badger, I don't know what else to use other than sincedb\_path for cloudwatch\_logs input plugin.

According to lukewaite, sincedb\_path can be used:

> **[lukewaite/logstash-input-cloudwatch-logs](https://github.com/lukewaite/logstash-input-cloudwatch-logs)**
>
> Input plugin for Logstash to stream events from CloudWatch Logs - lukewaite/logstash-input-cloudwatch-logs

Could you please tell me what should I use for http\_poller? Is that considered jdbc?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 26, 2019, 9:22pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443/4 "2019-06-26T21:22:21Z")

</div>

> [@EZprogramming](#):
>
> Could you please tell me what should I use for http\_poller? Is that considered jdbc?

An http\_poller input does not maintain state, and it is not jdbc.

---

<div class="post-metadata">

**Author:** ![EZprogramming](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ezprogramming/32/57291_2.png) [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\
**Post date:** [June 26, 2019, 9:25pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443/5 "2019-06-26T21:25:24Z")

</div>

I see, I guess I can just create document\_id for my output, so in that case I can overwrite the previous logs again with the same document\_id.

For more information:

> **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)**
>
> Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2019, 9:25pm UTC](https://discuss.elastic.co/t/what-logstash-input-plugins-can-you-set-sincedb-path-for/187443/6 "2019-07-24T21:25:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.