Require Help in understanding logstash

Elastic Stack Logstash
#1

Hi
I am new to logstash . I gone through some documentation about logstash and feel good to learn more about it.

The requirement which i am trying is .
To read the log file and mathc the keyword Access in the log file and display the lines

I have tried one sample conf file to read the log file and write the data which i search as output .

My conf file follows as like this

input
{
file {
path => ["/lee/logs/server.log"]
}
}
filter {
grok {
match => [ "message", "%{WORD:custom_name}" ]
}
}
output {
stdout {
codec => rubydebug
}

}

I verified by config it says OK

[lee@server1 logstash-1.5.0]$ bin/logstash -f test2.conf --configtest
Configuration OK
[lee@server1 logstash-1.5.0]$ bin/logstash -f test2.conf
Logstash startup completed
Access

I gave the input Access aftert this am not seeing any other thing.

Please help me if i am missing something to check this .

I Dont understand what is happening where i need to check as well after this

Thanks in Advance

Regards
Lee

#2

Has /lee/logs/server.log changed since you ran Logstash the last time?

#3

Hi Magnus

Thank you for your reply .

I checked the file and could see there is no change to my server.log

#4

Then there's nothing for Logstash to do. Logstash reads from the end of the file. If you want to reprocess old files you have to set start_position => beginning and either set sincedb_path => "/dev/null" or delete the sincedb file.

When playing around with Logstash I recommend that you use the stdin input instead and feed Logstash data via stdin.

#5

Hi Again

Correct me if i am wrong.

Stdin input is the input which will be passed in runtime . If i want to use the file how can i use it .
Could you refer me some paths about how to use stdin input to get the data from file

thanks in advance

#6

With the stdin input plugin you can pass data to Logstash in any of the following two ways:

/path/to/logstash ... < /path/to/logfile
cat /path/to/logfile | /path/to/logstash ...

This is obviously not specific to Logstash but how Unix-like shells work in general.

But again, this is rarely used for real work but it's quite useful if you want to play around without dealing with sincedb files etc.

#7

Hi Magnus

now my logstash config is reading the data of my log file and showing in the output . But with that i am getting the following information in my test as well
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["/lee/logs/server.log"], start_position=>"beginning", sincedb_path=>"/logs/", delimiter=>"\n">
Error: No such file or directory - /logs/.9788.32761.156686 {:level=>:error}

Could you please help me on this. where i am wrong what needs to look to fix this error as well
input
{
file {
path => ["/lee/logs/server.log"]
start_position => "beginning"
sincedb_path => "/logs/"
}
}
filter {
grok {
match => [ "message", "%{WORD:custom_name}" ]
}
}
output {
stdout {
codec => line
}

}

Thanks in Advance

#8

I found it My sincedb path is incorrect . I rectified it now and it is working.

Thank you for helpfule hints

#9

Hi..Less_ps
I am getting the same error while using sincedb..could you please tell how i can correct or set my sincedb path.

#10

Hi There,

Hope it will solve your problem
add sincedb_path to the path where you have read and write access for the file.
Example => "/home/user/Downloads/dbflile"
it will create one file with name dbfile and this file keeps track of the current position of monitored log files.

Exmple
input {
file {
path => "/home/user/Downloads/sample.csv"
type => "Sample"
start_position => "beginning"
sincedb_path => "/home/user/Downloads/dbflile"
}
}

//Lee

#11

Hi Lee_PS,

Thanks for the replay.
Actually i am using Windows OS so it will be different or it will be same.

Thanks,
Renu

#12

Hi,

Could you please check..this code
if ([message] =~ /(XAPMErrorEnrichmentXB :: Order line value is)/) {
grok {
match => {"message" => "(?(XAPMErrorEnrichmentXB :: Order line
value is ).{9})"}
add_field =>{ "orderId" => "%{result}" }
}
}

is it correct or not?

Thanks,
Renu

#13