Hi,
I want to parse a log but I am not able to achieve this. I am getting no output on my console. Whenever I run logstash using the command:
bin/logstash -f firstInput.conf --debug
I get this:
Starting pipeline {:id=>"main", :pipeline_workers=>8, :batch_size=>125, :batch_delay=>5, :max_inflight=>1000, :level=>:info, :file=>"logstash/pipeline.rb", :line=>"188", :method=>"start_workers"}
Pipeline main started {:file=>"logstash/agent.rb", :line=>"465", :method=>"start_pipeline"}
each: file grew: /Users/mac/project/Logstash/logs/access_logs.logs: old size 0, new size 94 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}
each: file grew: /Users/mac/project/Logstash/logs/access_logs.logs: old size 0, new size 94 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}
and this keeps on repeating.
I had written a custom regex for parsing this log which is stated below. I also checked it with grok debugger and it gave me the correct output.
"%{TIMESTAMP:timestamp} %{IPORHOST:clientip} %{IPORHOST:clientip2} %{IPORHOST:serverip} %{URI_FRONT:uri}"
Here:
URI_FRONT [0-9a-zA-Z*/][0-9a-zA-Z].[a-z][a-z][a-z]
TIMESTAMP (?>\d\d){1,2}-(?:0[1-9]|1[0-2])-(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
The config that I am using is:
input {
file {
path => "/Users/macproject/Logstash/logs/access_logs.logs"
sincedb_path => "/dev/null"
start_position => beginning
ignore_older => 0
}
}
filter {
grok {
patterns_dir => ["/Users/mac/project/Logstash/patterns"]
match => { "message" => "%{TIMESTAMP:timestamp}@~#%{IPORHOST:clientip}@~#%{IPORHOST:clientip2}@~#%{IPORHOST:serverip}@~#%{URI_FRONT:uri}" }
}
}
output {
stdout {
codec => rubydebug
}
}
The custom formats that I have given are being read by logstash but I am not able to get anything on my console.
Could you please help me. I am trying hard but unable to do it.