i use logstash for reading a log file .this file store my service log ,so added some logs during a hours. the point is each time that a log add in my log file , logstash read all log file from first and send it to elasticsearch . how can i fix this config that each time send just latest log to elasticsearch?
Can you show us your logstash input configuration?
This isn't the normal mode of work from logstash... The logstash store position read from each file and case you kill a process, the same start in correct position
Configure sincedb_path for the file input in logstash.
sincedb_path
Value type is string
There is no default value for this setting.
Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to <path.data>/plugins/inputs/file NOTE: it must be a file path and not a directory path
input {
file {
path => "/home/adanic/alternatives.log"
type => "log"
}
}
filter {
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "doc"
document_type => "%{type}"
}
}
it is my first config
i change my conf and set directly sincedb_path bud didn't work
hear my second conf
input {
file {
path => "/home/adanic/alternatives.log"
type => "log"
sincedb_path =>"/home/adanic/ee"
}
}
filter {
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "doc"
document_type => "%{type}"
}
}
Exactly how are you adding logs to the file? By appending to the file?
yes i append each log to end of the log file
How, exactly?
well ,i open log file and paste one row of log that i copied and save
In a text editor? Then you won't actually append to the file. The text editor will most likely create a new file, write the new data into that file, and rename it into place. To Logstash, this counts as a brand new file.
But the configuration you claim you have is inconsistent with the behavior you describe. Unless you set the file input's start_position option to "beginning" Logstash will under no circumstances read a file from the beginning (well, okay, unless the file is empty).
you mean i should set start_position => "end"?
No, I mean that you shouldn't use a text editor to simulate appending to a log file, because it won't actually be an append operation.
but i test it in windows and it was ok in this way of adding log.
ok i test it with system log file .this file i think use append operation for add log
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.