Fetching logs form remote host?

Elastic Stack Logstash
1

Hi,
I am using LSF as forwarder, logstash (as shipper and indexer on remote server) through redis.
We have some windows machines which have some log files to read. Is there any way so that I can groke those log files through shipper(logstash server).
These machines are nothing but devices and there number may change frequently and significantly in production. I don't want to install forwarder each time some device is added.
Please guide on this. I have refered logstahs book, but it suggests syslog configuration on remote machine, again tedious job.

br,
Sunil

2

I am using LSF as forwarder, logstash (as shipper and indexer on remote server) through redis.
We have some windows machines which have some log files to read. Is there any way so that I can groke those log files through shipper(logstash server).

If the logs are available via CIFS/SMB this should be doable, at least from a Windows machine. You should be able to point logstash-forwarder or Logstash to the remote log's UNC path (\\machinename\c$\path\to\file.log).

These machines are nothing but devices and there number may change frequently and significantly in production. I don't want to install forwarder each time some device is added.

If you don't have automation set up for this you're working too hard.

4

Hi Magnus,

I have log file on remote machine 10.xx.xxx.xxx and path is C:\var\log\common-logging\common-logging.log

I have configured path in logstash.conf as below.

path => "10.xx.xxx.xxx\c$\var\log\common-logging\common-logging.log"

Please let me know whats wrong in this. I am getting below error:

Plugin: <LogStash::Inputs::File type=>"applicationlogs_local", path=>["\\10.xx.xxx.xxx\c$\\var\\log\\common-logging\\common-logging.lo
g"], start_position=>"end">
Error: Neither current working directory (null) nor pathname (\10.xx.xxx.xxxc$\var\log\common-logging\common-logging.log) led to an absolute path
{:level=>:error}←[0m

5

Hi Magnus,
about your second comment: what kind of automation you are talking about?

6

path => "10.xx.xxx.xxx\c$\var\log\common-logging\common-logging.log"

That's not a valid UNC path. They always begin with \\. Also, I suspect that Logstash requires forward slashes instead of backslashes.

about your second comment: what kind of automation you are talking about?

You should automate how you set up machines so that installing and configuring additional software components isn't an issue.

7

Hi magnus,

I have give the remote path correctly even i am getting same error

It looks the text editor not taking double slash please consider this on each IP mentioned place in path and error message \\xxx.xxx.xxx.xxx
input {
file {
path => "\xxx.xxx.xxx.xxx\d$\logs\myserver.log"
start_position => "beginning"
}
}

filter {
grok {
match => { "timestamp" => "%{TOMCAT_DATESTAMP:timestamp}"}
match => { "level" => "[%{LOGLEVEL:level}]" }
match => { "class" => "[%{JAVACLASS:class}]" }
match => { "logmessage" => "%{JAVALOGMESSAGE:logmessage}" }
}
grok{
match => { "exceptions" => "%{JAVASTACKTRACEPART}" }
}
date {
match => [ "timestamp" , "yyyy-mm-dd HH:mm:ss Z" ]
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
}
stdout { codec => rubydebug }
}

Error while run logstash
D:\basefarm\logstash-2.0.0\bin>logstash -f logstashcfg.conf
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
io/console not supported; tty will not be manipulated
Default settings used: Filter workers: 1
←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["\\xxx.xxx.xxx.xxx/d$/logs/myserver.log"], start_position=>"beginning", codec=><LogStash::Code
cs::Plain charset=>"UTF-8">, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">
Error: Neither current working directory (null) nor pathname (\xxx.xxx.xxx.xxx/d$/logs/myserver.log) led to an absolute path {:level=>:
error}←[0m
Logstash startup completed
←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["\\xxx.xxx.xxx.xxx/d$/logs/myserver.log"], start_position=>"beginning", codec=><LogStash::Code
cs::Plain charset=>"UTF-8">, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">
Error: Neither current working directory (null) nor pathname (\xxx.xxx.xxx.xxx/d$/logs/myserver.log) led to an absolute path {:level=>:
error}←[0m

Please help me

8

Quoting myself:

Also, I suspect that Logstash requires forward slashes instead of backslashes.

So, have you tried with forward slashes?

9

Yes both I tried same error

D:\basefarm\logstash-2.0.0\bin>logstash -f logstashcfg.conf
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
io/console not supported; tty will not be manipulated
Default settings used: Filter workers: 1
←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["\\xxx.xxx.xxx.xxx\d$\logs\myserver.log"], start_position=>"beginning", codec=><LogStash::C
odecs::Plain charset=>"UTF-8">, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">
Error: Neither current working directory (null) nor pathname (\xxx.xxx.xxx.xxxd$logsmyserver.log) led to an absolute path {:level=>:err
or}←[0m
Logstash startup completed
←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["\\xxx.xxx.xxx.xxx\d$\logs\myserver.log"], start_position=>"beginning", codec=><LogStash::C
odecs::Plain charset=>"UTF-8">, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, delimiter=>"\n">
Error: Neither current working directory (null) nor pathname (\xxx.xxx.xxx.xxxd$logsmyserver.log) led to an absolute path {:level=>:err
or}←[0m

10

Hi,
I do this normally on our servers. this should work for you:

file {
 path => "\\\\******/d$/logs/myserver.log"
 start_position => "beginning"
 }
1 Like
12

I was getting the same error before, too. Simply changing backslashes to forward slashes worked great for me. It should be //xxx,x,x,x/e$/*.log