How to override @timestamp field by elasticsearch pipeline

Elastic Stack Elasticsearch
1

Hi guys

I use filebeat to collect logs and send to elasticesearch directly. And the log contain the time such as

2017-12-12 15:41:30.000 [pool-8-thread-18] DEBUG c.h.i.c.m.servicemanager.task.PassiveNodeHbTask - PassiveNodeHbTask started

So i want to use the time in the log to override @timestamp and i don't want to use logstash

I have difined a pipeline:

   PUT _ingest/pipeline/my-pipeline-05
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:timestamp}"]
      }
    }
  ]
}

and i use this pipeline, but it didn't override @timestamp it create another field name timestamp

here is json

{
  "_index": "tocc-2017.12.12",
  "_type": "doc",
  "_id": "rLe0SWAB2D7R7OG8M7Fc",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2017-12-12T07:50:33.475Z",
    "log": {
      "level": "temp",
      "source": "temp",
      "type": "JAVA LOG"
    },
    "prospector": {
      "type": "log"
    },
    "source": "D:\\Program Files (x86)\\iVMS8600-TOCC\\log\\cms\\debug.log",
    "message": "2017-12-12 15:50:31.045 [pool-8-thread-16] DEBUG c.h.cms.cache.core.memory.SimpleMemoryCache - get key [#0#service_info_cache_version],value is:d1f9cef1-c406-4c27-9d8e-efadb92d5a1a",
    "timestamp": "2017-12-12 15:50:31.045"
  },
  "fields": {
    "@timestamp": [
      "2017-12-12T07:50:33.475Z"
    ]
  },
  "sort": [
    1513065033475
  ]
}

So what can i do to override the @timestamp field

2

Use the date processor.

3

thanks i have already try ,and the elasticsearch and filebeat didn't show any errors

But the i could get any message if i use the pipeline below

PUT _ingest/pipeline/my-pipeline-05
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:logatime}"]
      },
      "date" : {
        "field" : "logatime",
        "target_field" : "@timestamp",
        "formats" : ["YYYY-MM-DD HH:mm:ss.SSS"]
      }
    }
  ]
}

And if i didn't use date processor

I could get the message.

So am i mistake something

4

You configuration parameters for the date filter does not seem to match what is described in the documentation, so I would recommend fixing that.

5

I didn't use logstash , in fact i use elasticsearch's pipelines and i follow the documentation.

6

Indeed. Not sure how I misread that. I would recommend that you compare the pattern against the docs and make sure all parts are specified using the correct case.

7

OK i will try ,and rewrite the date filter. I mean i didn't use logstash, and i misunderstand because i used believe maybe there have some different between the logstash's pipeline and elasticsearch's pipeline

8

Ok i find where is problem

Firstly, i need to use correct case

Second if you still could not find message , it could be you didn't set timezone, you need to set timezone and the problem would be solved

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.