What logstash input plugins can you set sincedb_path for?

Elastic Stack Logstash
1

I have seen some Logstash configuration online like this:

input {
  file {
    start_position => "beginning"
    sincedb_path => "/pathToSincedb/sincedb"
    path => "/pathToYourData/*"
  }
}

As we can see, there is the sincedb_path that is set for the File input plugin.

Question:

  1. What if the input plugin is http_poller or cloudwatch_logs, can I still use sincedb_path?
  2. If yes, what should I set my sincedb_path as if I am going to use a docker? With dockers, things are a bit different from usual path setup. I've heard I need to mount the path in my docker file.
  3. If no, how can I tell my program where it left off when reading the logs again?

Objective:

  • I understand that sincedb_path is by default set to null, but I don't want that. I want my program to keep track of what logs have been read so far, so the next time I restart my program again, it can start from where it left off.
  • Setup sincedb_path in my Logstash configuration that is used in a docker file.

I appreciate any feedback/suggestions.

2

No, by default the file input will generate a valid (but obscure) sincedb_path and persist the in-memory sincedb across restarts. Only a file input uses a sincedb to persist state. A jdbc input can use sql_last_value. It is possible that other inputs use other options, but I have no experience of those.

1 Like
3

@Badger, I don't know what else to use other than sincedb_path for cloudwatch_logs input plugin.

According to lukewaite, sincedb_path can be used:

Could you please tell me what should I use for http_poller? Is that considered jdbc?

4

An http_poller input does not maintain state, and it is not jdbc.

1 Like
5

I see, I guess I can just create document_id for my output, so in that case I can overwrite the previous logs again with the same document_id.

For more information:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.