Logstash not reading remote log files


(Surendrakumar Deshpande) #1

Hi,

I have some log files in a remote system, I need to send those to elasticsearch. But logstash is not reading the remote log files. I have specified the following configuration for logstash.

input {
file{
path => ["\\web1\C$\msu\logs\prod-hickory\prod-admin-debug.log","\\web1\C$\msu\logs\prod-hickory\prod-admin-error.log"]
start_position => "beginning"
sincedb_path => "demo3_since.db"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
output{
elasticsearch{
hosts => ["xxx.xx.xx.xx:9200"]
}
stdout{
codec => rubydebug
}
}

Can you please suggest if anything is wrong in this config file?


(Magnus Bäck) #2

You might want to try forward slashes instead of backslashes in the path. If that doesn't help, try increasing the logging verbosity by starting Logstash with --verbose or even --debug and look for clues.


(Mark Walkom) #3

Those look like Windows remote share paths, so \ is probably valid.


(Magnus Bäck) #4

Windows is fine with backslashes (and slashes), but I'm not sure Logstash is. I wouldn't trust its configuration lexer to not mangle them.


(Rene) #5

Are you sure there should be a \ between C and $? C$ seems to be more likely as it is the hidden share for the C-drive.


(Surendrakumar Deshpande) #6

Nothing is working, tried everything, do we have any other possible solution?


(Magnus Bäck) #7

Are the input files older than 24 hours? Then you need to adjust the file input's ignore_older option.

Also, did you try the suggestion I gave about starting Logstash with --verbose or --debug to get additional clues?


(Surendrakumar Deshpande) #8

Does it have anything to do with the authentication of the remote system?


(Magnus Bäck) #9

Possibly. If you start Logstash with --verbose it'll tell you if it doesn't have permissions to open a file. It'll also tell you if it's not able to find any files (which could also be caused by a lack of permissions) but that may require debug-level logging (i.e. starting with --debug), I don't remember OTOH.


(Surendrakumar Deshpande) #10

No such message was seen in the debug mode or verbose mode.


(Magnus Bäck) #11

So it does discover the files? There should be a log entry containing "discover" and the paths to the files you won't to monitor.

Also, what's in your sincedb file? If Logstash already think its current position is at the end of the file it obviously won't do anything.


(Netai Biswas) #13

I would like to know, how can I set the same path for linux system.


(Magnus Bäck) #14

@netaibiswas, please start a new thread for your unrelated question.


(system) #15