Extract special part from access.log file by Logstash

Dear Friends:
I want to use logstash to extract a special part from access.log file.For instance,to extract all the logs that @timestamp=="2017-10-24T16:41:16+08:00" from the target access.log file to a new file.That is to say “File Input to Filter to File Output”.
To make this come true,I installed logstash on the host where the access.log file exists.This logstash application is just as a sieve to select the special part of the access.log and make a new file to store them(the special part),that is to say the Input is access.log file,the Output is New file.
In the first step,I complete the installation of logstash and made a simple test by cmd "bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}' ",this turns out to be all right,it works well.
At the second step,I made a configuration file called logstash_core.conf, this is the core part to accomplish my original idea,by the certain plugins used or configured in this file to make the selection come true.
At the third step,after the configuration,I start the process of logstah by cmd "bin/logstash --verbose -f config/logstash_core.conf --config.reload.automatic &",but the wanted consequence does not appear.
I wonder if this(selection) is feasible.If it is ,where is the mistake in my operation?
The following if the configuration of my logstash application.

#input part
input{
file{
path => "/root/10.200.200.54_access.log" #the path to my access.log file
type => "web_log"
codec => "json"
start_position => "beginning"
}
}
#filter part
filter {
if [type] == "web-log" {
mutate {
gsub => ["url-info", "\x", "\\x"]
}
mutate {
gsub => ["agent", "\x", "\\x"]
}
}
}
filter{
date{
match => [ "@timestamp", "yyyy-MM-dd HH:mm:ss","ISO8601" ]
}
}
#output part
output{
if [@timestamp] == "2017-10-24T16:41:16+08:00" {
file{
path => "/root/target_file.log" ##the path to store my new file.
codec => "json"
}
}
}

Look forward to your response.
Regards.

  • The file input could be tailing the input file since it thinks it has already processed the file. Clearing the sincedb file or setting sincedb_path to /dev/null will address that.
  • Unless you're running Logstash as root it won't be able to access the input file.
  • Are you sure @timestamp will ever be equal to "2017-10-24T16:41:16+08:00"? I'm not so sure.

OK,Thank you at first for your time and energy.

List item

  1. I wonder if this is feasible(I mean Extract special part from access.log file by Logstash), make logstash as the filter just to select and save the wanted content of a access.log file? I need a sure answer.Or some constructive advices.
  2. About the "sincedb_path",I will follow your answer and try.
  3. Yes,I do it as the system user root,it is OK here.
  4. I thought @timestamp as a condition/variability ,so I can filter the special part message I want to save as a new file. This is a try,I wonder if it is OK.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.