Logstash replace existing fields & file input advice

Elastic Stack Logstash
1

Hi,

So this is a 2 part question. Below is an example of a log that is being read from a log file:

[09/10/2020 12:16:00] xxx-server - HTTP Connections Spiking Ok 3.00 Perf Counter test (Current Connections) 4857

The following is my current logstash config file (I have stripped it down to one log example to keep things simple as I can replicate the config from the solution anyone is able to provide me):

input {
	file {
		path => "C:/Temp/http.txt"
		type => "log"
		start_position => "beginning"
	}
}
filter {
	if [message] =~ "HTTP Connections Spiking"{
		grok {
			match => { "message" => "%{SYSLOG5424SD:Time}%{SPACE}%{HOSTNAME:hostname} - (?<event>\w+ \w+ \w+)%{SPACE}%{WORD:status}%{SPACE}%{NUMBER:reply}%{SPACE}%{GREEDYDATA:msg}" }
		}
		mutate {
			remove_field => [ "message" ]
		}
	}
}
output {
	elasticsearch {
		hosts => ["xxx-server:9200", "xxx-server2:9200", "xxx-server3:9200"]
	}
}

a) The input log file has a constant name, however after each month the current log file is renamed to (in this example) httpold.txt and remains in the same folder, a new http.txt file is then created and the logs for that month begin to populate.

Will logstash know this is a new file and simply continue looking at the latest record regardless of where it exists in the logfile, or will it have an issue that the log file has essentially started again?

b) how do I replace existing fields? I would like the replace the @timestamp field with the Time field that has been extracted from the grok expression. I have tried the match -> date as advised in other forums but get a parse error and not exactly sure how to match the date correctly.

I haven't got any further with the timestamp, but if someone is able to advise how I could also replace the host field with the hostname field I have extracted as well, that would be appreciated?

(I have not put my full config of the output section, as I know this already works from testing, just wanted to keep it simple).

thanks

Ian

2

The file input is designed to handle rotating log files. It should start tailing the new file when it is created, starting at the beginning. That said, spotting when a file has been rotated is an extremely complicated problem, and sometimes the file input gets it wrong. If the previous file is renamed then it will retain the inode number and that should allow the file input to correctly work out what is happening.

To parse the date try

date { match => [ "Time", "[MM/dd/YYYY HH:mm:ss]" ] }

(Or dd/MM if your month and day are the other way around.)

3

ah man,

thanks so much @Badger, I have tried so many combinations, and did get close to that but gave in. Helped me out loads there.

Thanks also for the file input advice.

Would you know how I would change the host field, so it overwrites the server that is providing the log with the server the log entry is relevant to?

thanks again

Ian

4

If you want to overwrite a field with the contents of another field you can use mutate+replace.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.