Filebeat 6.4.2 and 6.5.1: Read line error: "parsing CRI timestamp" and "invalid CRI log format"


(Raman Gupta) #1

I am getting various CRI parsing errors, on both Filebeat 6.4.2 and 6.5.1. I have cleared all filebeat state and restarted Filebeat, but these errors always occur. Here are a couple:

2018-12-04T02:32:18.636Z	ERROR	log/harvester.go:281	Read line error: parsing CRI timestamp: parsing time "3.chunk.js" as "2006-01-02T15:04:05Z07:00": cannot parse "unk.js" as "2006"; File: /var/lib/docker/containers/2a7d435274655d581cbdba63c8477042a30583c7cb84c9b2ef5a4c5af9bd2da7/2a7d435274655d581cbdba63c8477042a30583c7cb84c9b2ef5a4c5af9bd2da7-json.log

and

2018-12-04T14:26:29.853Z	ERROR	log/harvester.go:281	Read line error: invalid CRI log format; File: /var/lib/docker/containers/4da60db6721a545e448eb42732a57372d239b3882397a472e394504f9a448537/4da60db6721a545e448eb42732a57372d239b3882397a472e394504f9a448537-json.log

All the log lines from the input files referenced appear to be valid JSON, one JSON object per line.

It looks like these are not new -- looking through my logs, it appears these have been happening for some time, and on the previous 6.4.2 version of Filebeat as well, at a rate of around 6 / minute.

I use Filebeat in Kubernetes, with autodiscover enabled. Here is my config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true

    processors:
      - add_cloud_metadata: ~
      # This convoluted rename/rename/drop is necessary due to
      # https://discuss.elastic.co/t/filebeats-auto-mapping-of-kubernetes-labels-causing-big-issues/154718/9
      - rename:
          when:
            has_fields: ['kubernetes.labels.app.kubernetes.io/name']
          fields:
            - from: 'kubernetes.labels.app'
              to: 'kubernetes.labels.appobject'
          ignore_missing: true
          fail_on_error: false
      - rename:
          when:
            has_fields: ['kubernetes.labels.appobject']
          fields:
            - from: 'kubernetes.labels.appobject.kubernetes.io/name'
              to: 'kubernetes.labels.app'
            - from: 'kubernetes.labels.appobject.kubernetes.io/part-of'
              to: 'kubernetes.labels.part-of'
          ignore_missing: true
          fail_on_error: false
      - drop_fields:
          when:
            has_fields: ['kubernetes.labels.appobject']
          fields:
            - 'kubernetes.labels.appobject'

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:eslogging.default}:${ELASTICSEARCH_PORT:9200}']

(Raman Gupta) #2

Bump...


(Shaunak Kashyap) #3

You might be running into this bug: https://github.com/elastic/beats/issues/8203. Can you try the steps outlined in https://github.com/elastic/beats/issues/8203#issuecomment-422788000 and see if you still experience this issue?


(Raman Gupta) #4

Thanks for your reply. As per my OP, I noted : "I have cleared all filebeat state and restarted Filebeat". I still experience the issue.