Grok parser date processor problem

Elastic Stack Elasticsearch
1

Hi,
I have the grok parser below:

{
  "description": "pipeline",
  "processors": [
    {
      "grok": {
        "if": "ctx.message.contains('LOG1')",
        "field": "message",
        "patterns": [
          "\\[%{GREEDYDATA:Time}\\] \\(%{LOGLEVEL:Severity}\\)%{SPACE} (?<Module Name>%{WORD:ModulName}):  (?<Log Tag>%{DATA:logTag}) (?<Log Detail>%{GREEDYDATA:LogDetail})\\(%{GREEDYDATA:ErrorLineNumber}\\)"
        ]
      }
    },
	{
	  "date": {
        "field": "Time",
		"formats": ["yyyy.MM.dd HH:mm:ss.SSSSSS"],
		"timezone" : "Europe/Istanbul"
      }
	}
  ]
}

When the filebeat (v 7.9.2) started I show the logs:

|2020-10-24T16:45:13.807+0300|WARN|[elasticsearch]|elasticsearch/client.go:407|Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfdd29d9cc8bee8f, ext:1046063582, loc:(*time.Location)(0x607ff60)}, Meta:null, Fields:{"agent":{"ephemeral_id":"d0aa7d92-1001-4370-9256-2d9c06d41723","hostname":"pc","id":"62ab133e-e96b-47ca-84e9-d36936121c53","name":"pc","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"fields":{"apache":true,"applicationname":"App","logtype":"application","pipeline":"pipeline","type":"application"},"host":{"name":"pc"},"input":{"type":"log"},"log":{"file":{"path":"/home/dev/application/release/current_release/log/performance.log"},"offset":734},"message":" (main.cpp:223)"}, Private:file.State{Id:"native::8139491-64770", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc002307040), Source:"/home/dev/application/release/current_release/log/performance.log", Offset:756, Timestamp:time.Time{wall:0xbfdd29d9aee17d3d, ext:622096019, loc:(*time.Location)(0x607ff60)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x7c32e3, Device:0xfd02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"field [Time] not present as part of path [Time]"}|
|---|---|---|---|---|
|2020-10-24T16:45:13.807+0300|WARN|[elasticsearch]|elasticsearch/client.go:407|Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfdd29d9cc8c0562, ext:1046069425, loc:(*time.Location)(0x607ff60)}, Meta:null, Fields:{"agent":{"ephemeral_id":"d0aa7d92-1001-4370-9256-2d9c06d41723","hostname":"pc","id":"62ab133e-e96b-47ca-84e9-d36936121c53","name":"pc","type":"filebeat","version":"7.9.2"},"ecs":{"version":"1.5.0"},"fields":{"apache":true,"applicationname":"App","logtype":"application","pipeline":"pipeline","type":"application"},"host":{"name":pc"},"input":{"type":"log"},"log":{"file":{"path":"/home/test/application/release/current_release/log/performance.log"},"offset":464},"message":"[2020.10.16 16:38:35.044159] (ERROR)   Main:  \tipcs | grep [user_name] | awk '{ system (\"ipcrm -m \" $2)}' (cMemorySource.cpp:98)"}, Private:file.State{Id:"native::14157687-64770", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc002307520), Source:"/home/test/application/release/current_release/log/performance.log", Offset:593, Timestamp:time.Time{wall:0xbfdd29d9b3b389f7, ext:702970700, loc:(*time.Location)(0x607ff60)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xd80777, Device:0xfd02}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"unable to parse date [2020.10.16 16:38:35.044159] (ERROR)   Main:  \tipcs | grep [user_name]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020.10.16 16:38:35.044159] (ERROR)   Main:  \tipcs | grep [user_name] with format [yyyy.MM.dd HH:mm:ss.SSSSSS]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2020.10.16 16:38:35.044159] (ERROR)   Main:  \tipcs | grep [user_...' could not be parsed, unparsed text found at index 26"}}}|

I can't figure out this. Please help.

Regards.

3

Is the date unparseble with the parser below?

yyyy.MM.dd HH:mm:ss.SSSSSS

Any help?

4

Try this example

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "date": {
          "field": "Time",
          "formats": [
            "yyyy.MM.dd HH:mm:ss.SSSSSS"
          ],
          "timezone": "Europe/Istanbul"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "Time": "2020.10.16 16:38:35.044159"
      }
    }
  ]
}

does this work for you? If so, is anything missing compared to your example? Also, what Elasticsearch version are you using?

5

This works, and I use 7.9.2 version of elk stack.

6

can you provide a full reproduction, with an original logline plus your pipeline to reproduce locally?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.