Date filter not working for timestamps


(Shubham Munot) #1

Logs seems to be fine, yet date filter is not working. If i remove the date filter, it takes the timestamp as string, so grok pattern is fine.

Input format:

2017-04-03 11:40:59
2017-04-04 12:40:59
2017-04-05 13:40:59

Config file:

filter {
  if [message] =~ "^#" {
    drop {}
  }
  else {
     grok {
	match => {"message" => "%{TIMESTAMP_ISO8601:log_timestamp}" }	
	}

     date {
	match => ["log_timestamp","YYYY-MM-dd HH:mm:ss"]
	timezone => "Etc/UTC"
     }
  }
}

Logstash Logs:
[2017-05-25T10:55:42,070][WARN ][logstash.runner ] SIGTERM received. Shutting down the agent.
[2017-05-25T10:55:42,081][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-05-25T10:55:47,116][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{"other"=>[{"thread_id"=>28, "name"=>"[main]<beats", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.12-java/lib/logstash/inputs/beats.rb:213:in run'"}, {"thread_id"=>25, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:136:insynchronize'"}], ["LogStash::Filters::Mutate", {"remove_field"=>["log_timestamp"], "id"=>"1980d2e3a0a9ae26700c4a8ac71d34c4e8e536b8-8"}]=>[{"thread_id"=>23, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:136:in synchronize'"}, {"thread_id"=>24, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:118:insynchronize'"}, {"thread_id"=>26, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:118:in `synchronize'"}]}}
[2017-05-25T10:55:47,121][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2017-05-25T10:56:03,360][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-05-25T10:56:03,365][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-05-25T10:56:03,507][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x4bca867 URL:http://localhost:9200/>}
[2017-05-25T10:56:03,510][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x4caeb5ce URL://localhost:9200>]}
[2017-05-25T10:56:03,610][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-05-25T10:56:04,278][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2017-05-25T10:56:04,327][INFO ][logstash.pipeline ] Pipeline main started
[2017-05-25T10:56:04,382][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-05-25T10:56:08,598][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[http://localhost:9200/], :added=>[http://127.0.0.1:9200/]}}
[2017-05-25T10:56:08,598][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
[2017-05-25T10:56:08,604][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x77264f91 URL:http://127.0.0.1:9200/>}


(Anh) #2

Have you tried to output to stdout first instead of to Elasticsearch? Your filters look ok to me.


(Shubham Munot) #3

Hey, it was working. I just forgot to change the date filter on Kibana. I was just looking at last 15 minutes while those logs were a month old. So everything is working now.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.