Timestamp error

[space.iitr (Amit Sharma)]

I am getting wrong value in @timestamp field for elasticsearch/filebeat. My filebeat pipeline definition

curl -H 'Content-Type: application/json' -XPUT "logger:9200/_ingest/pipeline/app_log" -d'
{
  "description" : "Ingest pipeline for Jetty server log",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:timestamp} (%{UUID:accessid})? \\[(?<threadname>[^\\]]+)\\] %{LOGLEVEL:level} %{DATA:classname} - %{GREEDYDATA:message}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": [ "yyyy-mm-dd H:m:s,SSS" ]
      }
    }
  ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}'

Simulation result for some sample logger line. ( Using logback configuration )

curl -H 'Content-Type: application/json' -XPOST "logger:9200/_ingest/pipeline/app_log/_simulate?pretty" -d'
{
"docs": [
{
"_source": {
"message": "2018-03-17 22:38:39,079 bab3157d-a11c-4dba-a6d6-c47ae0de2b7f [qtp224100622-174782] INFO i.n.core.services.cache.CacheBuilder - Key : ChIJrTTTJkdsrjsRXkrYRKRRfd8-seo-localitiesv1 is returned from cache"

  }
},
{
  "_source": {
    "message": "2017-12-12 01:14:12,079  [qtp224100622-185269] WARN  i.n.m.cache.sdk.RedisCacheProvider - No matching policy: class in.nobroker.core.domain.Token"
  }
}

]
}'
The result from this simulation:

{ "docs" : [ { "doc" : { "_index" : "_index", "_type" : "_type", "_id" : "_id", "_source" : { "accessid" : "bab3157d-a11c-4dba-a6d6-c47ae0de2b7f", "@timestamp" : "2018-01-17T22:38:39.079Z", "classname" : " i.n.core.services.cache.CacheBuilder", "level" : "INFO", "message" : "Key : ChIJrTTTJkdsrjsRXkrYRKRRfd8-seo-localitiesv1 is returned from cache", "timestamp" : "2018-03-17 22:38:39,079", "threadname" : "qtp224100622-174782" }, "_ingest" : { "timestamp" : "2018-03-17T15:35:35.543Z" } } }, { "doc" : { "_index" : "_index", "_type" : "_type", "_id" : "_id", "_source" : { "@timestamp" : "2017-01-12T01:14:12.079Z", "classname" : " i.n.m.cache.sdk.RedisCacheProvider", "level" : "WARN", "message" : "No matching policy: class in.nobroker.core.domain.Token", "timestamp" : "2017-12-12 01:14:12,079", "threadname" : "qtp224100622-185269" }, "_ingest" : { "timestamp" : "2018-03-17T15:35:35.543Z" } } } ] }

Please notice that @timestamp field is totally different from timestamp field.

Incorrect timestamp by date processor in elasticsearch filebeat

[tudor (Tudor Golubenco)]

You seem to parse the timestamp field but never put it in @timestamp? Try adding "target_field": "@timestamp". Here is an example from our modules.