Doubts about creating indexes based on log timestamps


(Luis Cañas Díaz) #1

Hi guys!

during the last days I've been reading documentation in order to get my apache logs imported in different indexes based on the date of the log entries. With the configuration below I got it running with indexes like "index_period_2016dec". Now I would like to go further and get the number of the month instead of the name, so it is easier to sort the different indexes.

My two questions are:

  • is "correct" the way I did it with the whole regexp for grok? are there any other ways to do this in a simpler way?
  • how do I got the month (string) translated to its number?
filter {
    if [type] == "apache-access" {
        grok {
	    match => [ "message", '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[(?<timestamp>%{MONTHDAY}/%{MONTH:auxmonth}/%{YEAR:auxyear}:%{TIME} %{INT})\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}']
        }
        date {
            locale => "en"
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
	mutate {
	    lowercase => [ "auxmonth" ]
	}
    }
}

output {
    if [request] =~ /(\.ova)|(\.qc2)|(\.rpm)|(\.gz)|(\.bz2)|(\.deb)$/ {
	if [response] =~ /^2\d\d/ {
        	elasticsearch {
	            hosts => ["elasticsearch:443"]
        	    ssl => true
	            ssl_certificate_verification => false
	            path => "data"
	            user => "myuser"
	            password => "****"
	            index => "index_period_%{auxyear}%{auxmonth}"
	            manage_template => false
	            workers => 5
	            document_type => "items"
	        }
	}	
    }
}

(Christian Dahlqvist) #2

The usual way to create time-based indices from Logstash is to base the index name on the @timestamp field. An example of how to do this is the default index pattern for the index parameter. If you changed your index parameter to index => "index_period_%{+YYYY.MM}" you would get a monthly index with numeric month.


(Luis Cañas Díaz) #3

Thanks for your reply Christian :). As far as I know by doing that you get a numeric month based on the current time. So this is not what I'm looking for.


(Christian Dahlqvist) #4

It will give you numeric month based on the @timestamp field, which gets populated by your date filter based on the value in the 'timestamp' field.


(Luis Cañas Díaz) #5

You are right. I did some tests a days ago and I thought it did not work but it does!!

Thanks for the support :+1:


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.