Overwriting @timestamp in module

[mcardlesam (Sam)]

I have written a module for filebeat to parse a log from a application I have written

The pipeline mostly works, except in kibana i get the error "Text '19/May/2020 15:25:17' could not be parsed at index 12" for each record

And example log entry looks like
[19/May/2020 15:25:17] INFO [module name] the log message

My pipeline file looks like

    {
      "description": "Pipeline for parsing mylogs.",
        "processors": [
          {
            "grok": {
              "field": "message",
              "ignore_missing": true,
              "patterns": [
                  "\\[%{DATETIME:my.datetime}\\] %{LOGLEVEL:level} \\[%{GREEDYDATA:module}\\] %{GREEDYDATA:message}"
              ],
              "pattern_definitions": {
                "DATETIME": "%{MONTHDAY:day}/%{MONTH:month}/%{YEAR:year} %{TIME:time}",
                "GREEDYDATA": "(.|\n|\t)*"
              }
            }
          },
      
          {
            "date": {
              "field": "my.datetime",
              "target_field": "@timestamp",
              "formats": ["dd/MMM/yyyy HH:mm:ss"]
            }
          }
        ],
        "on_failure": [
          {
            "set": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }

In kibanna the @timestamp is displayed as when the log was parsed by filebeat, not the timestamp of the entry.
The other fields are all correctly shown in the record (ie day, month, year.. and the full datetime)
In the record there is the error.message filed with Text '19/May/2020 15:25:17' could not be parsed at index 12

I think that error message means index 12 of my format string causes an error when parsing the input? So this would be the hour?

[mtojek (Marcin Tojek)]

You can try to use the Grok Debugger: https://grokdebug.herokuapp.com/

[mcardlesam (Sam)]

The grok pattern works fine - the timestamp is correctly parsed out of my log,

The issue is overriding the @timestamp filed. This should have the my.datetime value, but it does not

[nmoham (Naveed)]

seeing a similar issue - Custom filebeat module timezone conversion issue

In my case, the timestamp is parsed and set, but the timezone always off by -8 hrs, when viewing data in Kiaban. This is definitely not related to Kibana related settings, but something inside filebeat modules.

[mcardlesam (Sam)]

Thanks @nmoham, reading you post helped, didnt realize I'd have to delete the pipeline in ES when changing the pipeline.yml file in filebeats.

I had previously fixed a mistake in the timestamp format but that was not reflected in the pipeline in ES.

To fix a used the dev-tools to run

DELETE /_ingest/pipeline/filebeat-7.7.0-my-log-pipeline 

And reparsed my logs - now the @timestamp filed is correctly set

[nmoham (Naveed)]

@mcardlesam Glad, it helped you . I am still stuck, I think the problem is when my module is sending the event, it lands into ES with UTC timezone.

"message": "May 19 16:31:52 syslog tmm2[18722]: Rule /Common/IP_Reputation_Blocking_iRule <CLIENT_ACCEPTED>: Dropping request. VS IP: 10.xxx.xx.xx, Client IP: xxx.xx.xxx.xxx, ...., "@timestamp": "2020-05-19T16:31:52.000Z", "ecs": { "version": "1.5.0" },

Looked at other logs in our environment, the expectation is -

"message": "May 19 16:31:52 syslog tmm2[18722]: Rule /Common/IP_Reputation_Blocking_iRule <CLIENT_ACCEPTED>: Dropping request. VS IP: 10.xxx.xx.xx, Client IP: xxx.xx.xxx.xxx, ...., "@timestamp": "2020-05-19T23:31:52.000Z", "ecs": { "version": "1.5.0" },

Also , I am routing logs to a different index using the date index Name processor.