Filebeat cannot parse a custom @timestamp string

[mattia_galati (Mattia)]

Hello everyone,
I tryed to search in several topics but cannot manage to find a solution.

I have a Python application that is writing logs and a Logstash process which is picking up that logs to send them to an Elastic index. Now we want to switch to Filebeat but we are conducting some tests in order to be sure to not lost anything.

The log is in JSON format, something like this:

{"@timestamp":"2020-10-20T06:53:09.324287","level":"INFO", "message": "My message"}

The @timestamp field itself is the string that comes out from the Python method

datetime.utcnow().isoformat()

Now, in my Filebeat config file i put the json.overwrite_keys: true directive because I noticed that the @timestamp on Elastic didn't reflect the actual timestamp in the log.
Once I activated that directive, Filebeat was no more able to parse the logs, saying:

Won't overwrite @timestamp because of parsing error: parsing time "2020-10-22T08:53:45.256544" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "Z07:00"

Then I stumbled upon the "timestamp" processor and and tried to parse the field like this:

- timestamp:
      field: "@timestamp"
      layouts:
        - '2006-01-02T15:04:05.000000'
      test:
        - '2020-10-22T08:35:40.415958'

The tests is passed and the process manage to start but it seems like ignores it and keep raising that error when collecting logs.
What I'm doing wrong?

[mattia_galati (Mattia)]

Hi there,
waiting for a more accurate reply, for those interested or stucked on same problem, I decided to try a different approach and surprisingly it worked. (You must have Docker installed)

1. Download the source code of Beats publicly available on Github
2. Extract in a folder of your choice
3. Change the file libbeat\common\jsontransform\jsonhelper.go at line 54, replacing the RFC3339 with a parse format of your choice
4. Create a new Dockerfile in the root of "beats" folder with this content:

FROM golang:1.15.3-alpine3.12 AS build

RUN apk update && \
    apk add --no-cache bash git make musl-dev build-base && \
    export GOPATH=/root/go && \
    export PATH=${GOPATH}/bin:/usr/local/go/bin:$PATH && \
    export GOBIN=$GOROOT/bin && \
    mkdir -p ${GOPATH}/src ${GOPATH}/bin && \
    export GO111MODULE=on && \
    go version && \
    git clone https://github.com/magefile/mage.git && \
    cd mage && \
    CGO_ENABLED=1 GOOS=linux go run bootstrap.go

COPY . /opt/beats/

RUN cd /opt/beats/filebeat && mage build

FROM alpine:3.12

COPY --from=build /opt/beats/filebeat/filebeat /usr/share/filebeat/filebeat
COPY --from=build /opt/beats/filebeat/filebeat.reference.yml /usr/share/filebeat/filebeat.reference.yml
COPY --from=build /opt/beats/filebeat/filebeat.yml /usr/share/filebeat/filebeat.yml
COPY --from=build /opt/beats/filebeat/module /usr/share/filebeat/module
COPY --from=build /opt/beats/filebeat/modules.d /usr/share/filebeat/modules.d

ENV PATH="/usr/share/filebeat:${PATH}"
WORKDIR /usr/share/filebeat

ENTRYPOINT ["filebeat"]

5. Now run docker build -t [your_image_name] -f [your_dockerfile_name] . (don't forget the dot at the end)

Now you have an image of a custom Filebeat, with your parser.

Enjoy.