Error while uploading a Logfile using Logstash

Elastic Stack Logstash
1

Hi All,

I have simple logs in the format:
<[package.filename]>

The problem I faced is that the Logfile sometimes fails to get uploaded and i have to make changes in the file to get it uploaded. Moreover capital letters are not allowed in naming of indexes which I'm not sure why.

The content of the script file I used:
input {
file {
path => ["D:\logs\Log1.txt"]
start_position => "beginning"
type=> "log"
}
}

filter {

}

output {
elasticsearch {
action => "index"
index => "app1-2018-06-11"
hosts => ["localhost:9200"]
document_type => "log"

}

}

2

The problem I faced is that the Logfile sometimes fails to get uploaded and i have to make changes in the file to get it uploaded.

Logstash remembers how far it has processed a file so if you need to reprocess a file you must clear its sincedb entry or set the file input's sincedb_path option to "nul" (WIndows) or "/dev/null" (non-Windows). See the file input documentation.

3

Thanks for the reply.
I just want to know that if I delete all the sincedb files will it make any difference apart from reparsing the log files again when I run logstash?

4

No, that's the only effect.

5

Hey Magnus baeck,

Thanks for the help but I'm facing another issue i.e. the log file needs to be modified a little(anything like a simple newline or deleting a line) and only then does Logstash upload it.
Any help regarding this?

6

It sounds like you didn't actually clear the sincedb state.

7

I have added the sincedb_path to null in the conf(script) file and removed the older sincedb files as well.

8

I have added the sincedb_path to null

"null" or "nul"?

To debug what's going on when Logstash is reading file you can increase the loglevel.

9

this is the command written in input part of script:

sincedb_path => "null"

the location of these files is:
logstash-6.2.4\data\plugins\inputs\file

Is the location correct for the sincedb files?
If yes then I've tried many times and no files are present in this folder.

10

sincedb_path => "null"

This is wrong and will write the sincedb data to a file named "null". Use "nul" instead.

11

Oh!
Thanks I'll try that.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.