Logstash date filter in string format for input 150M

[pcouas]

Hi

My logstash filter logdate is converted to "string" field and not to "date" field, if all my log rotated field are more 150 Mo ?
An idea ?

Regards
Phil
logstash 5.6.8

[magnusbaeck (Magnus Bäck)]

if all my log rotated field are more 150 Mo ?

I have no idea what you mean by this.

What does your configuration look like? What does an example event produced by Logstash look like? Use a stdout { codec => rubydebug } output.

[pcouas]

Hi

My logstash configuation file is

input {
file {
type => "static"
path => "/home/elasticsearch/static_logs/Web.log" exclude => ".zip"
start_position => beginning
sincedb_path => "/dev/null"
}
}

filter {
if [type] == "static" {
if [message] !~ /(.+)/ {
drop { }
}
grok{
patterns_dir => "./patterns"
overwrite => [ "message" ]
# 2017-08-07 11:47:35,466 INFO [http-bio-10.60.2.19-10267-exec-60] jsch.DeployManagerFileUSImpl (DeployManagerFileUSImpl.java:155) - Deconnexion de l'hote qvirmqa3
# 2017-08-07 11:47:51,775 ERROR [http-bio-10.60.2.19-10267-exec-54] service.BindingsRSImpl (BindingsRSImpl.java:143) - Can't find bindings file deployed on server
# 2017-08-03 16:01:11,352 WARN [Thread-552] pcf2.AbstractObjetMQDAO (AbstractObjetMQDAO.java:137) - Descripteur de
match => [ "message", "%{TIMESTAMP_ISO8601:logdate},%{INT} %{LOGLEVEL:logLevel} [(?[^]]+)] %{JAVACLASS:package} (%{JAVAFILE:className}:%{INT:line}) - %{GREEDYDATA:message}" ]
}
# 2017-08-03 16:01:11,352
date{
match => [ "logdate", "YYYY-MM-dd hh:mm:ss" ]
target => "logdate"
}
}
}

output {
elasticsearch { hosts => ["192.168.99.100:9200"]}
}

Regards

[Badger]

If logdate starts as the string "2017-08-07 11:47:51" then you get

       "logdate" => 2017-08-07T15:47:51.000Z,

However, if it starts as "2017-08-07 16:01:11" then you get

       "logdate" => "2017-08-07 16:01:11"

Update your date filter to use HH instead of hh.

date { match => [ "logdate" , "YYYY-MM-dd HH:mm:ss" ] target => "logdate" }

[pcouas]

Thanks