jukkendar
August 10, 2016, 2:17pm
1
I have installed Logstash 2.3.1, elastic search 2.3.1 and Kibana 4.5.0 in windows 2008 server as services.
input {
path => "E:/logstash/New/Logsetup/log/system.day20160518.log"
start_position => "beginning"
type => "logs"
}
}
filter {@timestamp }" ]
}
output {
This line should not be in your elasticsearch output block configuration, and should actually result in an error (at the very least, a deprecation message):
jukkendar:
type => "logs"
And these lines are unnecessary as they are the default values:
Also, if you ran with this statement:
more than one time, it will not re-read the file, even with start_position => "beginning". This is because there is a sincedb file created that wants to tail and resume where it left off. For re-reading old log files that will not receive any more data, you can get them to reread by adding sincedb_path => NUL (NUL is like /dev/null for Windows machines).
jukkendar
August 10, 2016, 3:40pm
3
theuntergeek:
sincedb_path =>
Thanks for your reply. I have altered the config file as below but no go.
input {
path => "E:/logstash/New/Logsetup/log/system.day20160518.log"
sincedb_path => "/dev/null"
start_position => "beginning"
type => "logs"
}
}
filter {@timestamp }" ]
}
output {
}
}
As mentioned, /dev/null is for UNIX systems. You have a Windows path, so you should be using sincedb_path => NUL.
You should also test the output by changing the output block to be:
output {
stdout { codec => rubydebug }
# elasticsearch { }
}
With the elasticsearch output disabled, by being commented. You will be able to see output at the command-line this way. If you do not get any output, that will also indicate why the index is not being created.
jukkendar
August 10, 2016, 4:05pm
5
When I try debugging in command line , getting the below message.
E:\logstash\New\logstash-2.3.1\bin>logstash -f E:\logstash\New\logstash-2.3.1\bi
T_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:na
"62", :method=>"add_pattern"}←[0m
This seems incomplete. The Adding pattern lines aren't helpful and do not indicate any errors. Can you filter those out?
and in the same grok block:
Is StartTime getting its data from grok, or from @timestamp?
jukkendar
August 10, 2016, 4:59pm
8
App log file will look like , so used grok timefield based on log.
2016/08/09 00:43:38.169 S-300000 text text : ssytem error.
Updated conf without grok and getting the below error
input {
path => "E:/logstash/New/Logsetup/log/system.day20160518.log"
sincedb_path => "Nul"
start_position => "beginning"
type => "logs"
}
}
output {elasticsearch
←[36mconfig LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"/l@ignore_older = 86400 {:level=>:debug, :file=
"/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-@close_older = 3600 {:level=>:debug, :file=>"
If the modification time of system.day20160518.log is older than 24 hours you need to adjust the file input's ignore_older option.
jukkendar
August 10, 2016, 5:55pm
10
I tried after changing the DATESTAMP with current date as a file name and also the DATESTAMP
Even though i am getting a same error as i posted below:
E:\logstash\New\logstash-2.3.1\bin>logstash -f logstash.json --debug@charset = "UTF-8" {:level=>:debug, :file=>"@path = ["E:/logstash/New/Logsetup/log/system@sincedb_path = "Nul" {:level=>:debug, :file=
"/logstash/New/logstash-2.3.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-@start_position = "beginning" {:level=>:debug
I wouldn't even bother with grok before ensuring that data flows. You can comment out the entire filter block with # on each line. If you don't see data flowing, chasing grok is not going to help.
Try the ignore_older suggestion from @magnusbaeck . I'm also not sure whether Nul and NUL are the same to Windows, so I suggest making Nul into NUL.
jukkendar
August 15, 2016, 9:25am
12
I have my log file name like 00331_ln2_systemday20160814 , so when creating a index I want to add 2 new fields like store no from file name "00331" and Laneno as "ln2". Refer my grok file below.
filter {
You'll find the path to the file in the path field, so just add a grok filter that extracts the desired fields from that field.
jukkendar
August 15, 2016, 11:04am
14
Thanks magnus for quick reply. Do you mean the below line is correct? . Path has full file path, so is there any param for filename?
Please share me some example.
add_field => {"Store_num","%{GREEDYDATA}/%{GREEDYDATA:path}"}
Do you mean the below line is correct?
No. For starters it's not a grok filter.
Path has full file path, so is there any param for filename?
grok {
match => {
"path" => "/(?<filename>[^/]+)$"
}
}
I gave this exact example and explained how it worked in another thread just a few days ago.