Logstash since_db => "NUL" not forcing reload

Elastic Stack Logstash
1

Hello all,

I've built a ELK stack that loads a CSV on an hourly basis. The CSV is of all our tasks from jira, and so every change in an issue gets updated in the CSV, and needs to be reloaded to Kibana.

The stack works fine on loading, every new row gets sent, but logstash doesn't reload the entire file unless i manually restart it. I've defined the start point and sincedb_path correctly to my knowledge.

Server - windows
Version - 6.1.1

My input:

input {
  
  file {
  path => "C:/CFS/ELK/jira_raw.csv"
   start_position => "beginning"
   sincedb_path => "NUL"
   codec => plain {
                    charset => "UTF-8"
            }
  }
}

I've also tried with "/dev/null" for the heck of it even though it's a windows server

If anybody knows what might be the reason it dosen't work, or how i can time a reset of logstash, it would be highly appreciated.

All the best,

Noam

2

Logstash since_db => “NUL” not forcing reload

That is correct. Setting 'sincedb_path => “NUL”' prevents the in-memory sincedb being persisted across restarts, but the file input still maintains the sincedb and there is no way to stop that.

Can you include the date/time in the filename when you extract tasks from jira?

3

Do you mean to the CSV I export?
Yes, i can, would that solve it? That would require me to change the path to be
path => "C:/CFS/ELK/*.csv"
correct?

Thank you,

Noam

4

Correct.

5

Solved. Thank you!
For some Reason it isn't loading the final row, but probably a matter of config that isn't relevant to the issue.

Thanks

6

Does the last line end with a newline?

7

Yes, I can remove it but it didn't seem to be a problem

8

I was thinking the other way around. If the last line did not have a newline at the end then the file input would continue tailing the file, waiting for the newline to appear so that it could ingest the last line.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.