I'm trying to import log files from kiwi to elasticsearch using logstash. These log files are ANSI encoded with each log entry separated by a newline.
2019-05-24 11:23:25 Daemon.Notice WIN-DLI9AO07ERU May 24 11:23:24 WIN-DLI9AO07ERU Service_Control_Manager: 7036: The Windows Update service entered the running state.
That is an example of the logs I'm dealing with, and this is the config file I created for it.
input {
file {
file_sort_by => "last_modified"
mode => "tail"
path => "/home/elkbnk/Kiwi Log Files/*.txt"
start_position => "beginning"
codec => line {
charset => "Windows-1252"
}
type => "kiwisyslog"
tags => ["_kiwisyslog"]
sincedb_path => "/home/elkbnk/Kiwi Log Files/.sincedb"
}
}
filter {
grok {
match => {
"message" => ["%{SYSLOGBASE5}\s%{GREEDYDATA:[kiwi][syslog][message]}"]
}
pattern_definitions => {
"FACILITIES" => "(\w+)"
"SYSLOGBASE5" => "%{TIMESTAMP_ISO8601:@timestamp}\t%{FACILITIES:[kiwi][syslog][facility]}.%{LOGLEVEL:[kiwi][syslog][level]}\t%{SYSLOGHOST:[host][name]}\t%{SYSLOGTIMESTAMP:[kiwi][sysl$
}
tag_on_failure => "_grokparsefailure"
}
date {
match => [ "@timestamp", "yyyy-mm-dd HH:mm:ss" ]
tag_on_failure => "_dateparsefailure"
}
}
output
{
file{
path => "/tmp/logstashfiletest.txt"
}
if "_grokparsefailure" not in [tags]
{
if "_dateparsefailure" not in [tags]
{
elasticsearch
{
hosts => ["https://10.21.112.119:9200"]
user => "logstash_internal"
password => Whoops, nothing to see here
ssl => true
cacert => "/etc/certs/elastic-stack-ca.pem"
}
} else
{
file
{
path => "/var/log/parsefailure/dateparsefailure.log"
}
}
} else
{
file
{
path => "/var/log/parsefailure/grokparsefailure.log"
}
}
}
I used to have a check before the filter and output to see if it should go through that processing, but I removed it and all other inputs for testing purposes. That is also the only config file I have in the directory. No output files are made whether they are the parsefailure logs nor the test output file. It is also not showing up in kibana.
The debug logs show that the files are getting inputted line by line:
[2019-06-12T15:11:41,381][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/elkbnk/Kiwi Log Files/SyslogCatchAll-2019-04-01.txt", :text=>"2019-04-01 00:50:24\tDaemon.Notice\tWIN-DLI9AO07ERU\tApr 1 00:50:23 WIN-DLI9AO07ERU Service_Control_Manager: 7036: The WinHTTP Web Proxy Auto-Discovery Service service entered the stopped state.\r"}
Before I moved the other config file, filebeat data was flowing through fine, I just can't get the logs to work.
Any help would be greatly appreciated and I'd be happy to provide more logs and info if you need it.
