I've been trying to upgrade to the latest versions of ELK on a windows box to no avail.
Testing my configurations one piece at a time starting with logstash.
Current version of logstash that works with my configurations is 1.5.3
I downloaded logstash 2.2.2 zip for windows from the download page.
I copied my configurations over from 1.5.3 logstash directory. I have a config directory with the configurations broken out into different files: 1 file for inputs, 1 file for each filter, and 1 file for outputs.
I setup the input to read a test file, and the output to write to a file so I can verify things look correct, but it appears that logstash isn't even reading the input file. After logstash starts up I see no activity in my output directory until I hit CNTRL-C to stop logstash then the since_db file is created with no data in the file.
Here is my input configuration:
input {
file {
path => "D:/temp/log_stash/testfile.txt"
type => "TG"
start_position => "beginning"
sincedb_path => "D:/temp/log_stash/tg_sincedb"
}
}
Output configuration:
output {
if [type] == "TG" {
file {
path => "D:/temp/log_stash/tg_results.txt"
flush_interval => 0
}
}
}
Filter for my custom log file:
filter {
if [type] == "TG" {
mutate {
gsub => [ "message", "\r", "" ]
}
grok {
match => ["message", "\[%{TIMESTAMP_ISO8601:LogTimestamp}\] %{WORD:Status}-%{GREEDYDATA:RepId}, %{GREEDYDATA:Test}, Session:%{GREEDYDATA:Session}, package id: %{UUID:ClientPackageIdentifier}, %{TIMESTAMP_ISO8601:SentAt}, (?<ClientIdentifier>\S*)",
"message", "\[%{TIMESTAMP_ISO8601:LogTimestamp}\] %{WORD:Status}-%{GREEDYDATA:RepId}, Session:%{GREEDYDATA:Session}, package id: %{UUID:ClientPackageIdentifier}, %{TIMESTAMP_ISO8601:ReceivedAt}, Sequence:%{NUMBER:Sequence}, RequestRetries:%{NUMBER:RequestRetries}, ResponseRetries:%{NUMBER:ResponseRetries}, %{WORD:ResponseReason}, %{WORD:ErrorCode}, Occured at: %{TIMESTAMP_ISO8601:OccuredAt}, SSI:%{POSINT:SSI}, ClientId: %{GREEDYDATA:ClientIdentifier}, RadioType: %{GREEDYDATA:RadioType}",
"message", "\[%{TIMESTAMP_ISO8601:LogTimestamp}\] Response-%{DATA:Status}, Radio id:%{GREEDYDATA:RepId}, Package id:(?<ClientIdentifier>\S*)"
]
add_tag => [ "%{ClientIdentifier}" ]
add_tag => [ "%{Status}" ]
}
date {
match => ["LogTimestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
}
#if "_grokparsefailure" in [tags] {
# drop { }
#}
}
elapsed {
start_tag => "Sending"
end_tag => "Response"
unique_id_field => "ClientPackageIdentifier"
timeout => 61
new_event_on_match => false
}
}
Command line output:
D:\elk\logstash-2.2.2\bin>logstash.bat agent -f ../configs
io/console not supported; tty will not be manipulated
Settings: Default pipeline workers: 8
←[33mDefaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker th
eads {:count_was=>8, :filters=>["multiline", "multiline", "multiline", "multiline"], :level=>:warn}←[0m
Logstash startup completed
^←[33mSIGINT received. Shutting down the pipeline. {:level=>:warn}←[0mC
Terminate batch job (Y/N)? Logstash shutdown completed
y
Sample log lines:
[2016-01-06 21:30:56.620] Sending-012372000003227, Class 1 Poll, Session:0, package id: c4916ca2-cbfb-446a-91d8-53b11eeb1b9e, 2016-01-06 21:30:56.600, WIN7X64-VM-LAB-TG-20160106-213038
[2016-01-06 21:31:01.334] Response-012372000003227, Session:0, package id: c4916ca2-cbfb-446a-91d8-53b11eeb1b9e, 2016-01-06 21:31:01.325, Sequence:37, RequestRetries:0, ResponseRetries:0, NORMAL, NO_ERROR, Occured at: 2016-01-06 21:30:59.155, SSI:94, ClientId: WIN7X64-VM-LAB-TG-20160106-213038, RadioType: cellular
[2016-01-06 21:31:01.339] RTT-012372000003227, Session:0, 2016-01-06 21:31:01.325, 4.7244
Any help would be appreciated.
Thanks!