Elasticsearch do not create index from Filebeat

Hello,
I have a problem in creating index json in filebeat. It can not parse log file.

This is my log file:

This is my filebeat config:
Screen Shot 2022-05-04 at 00.32.14

This is announcement from filebeat:

How can I solve this ? Please.
And I want to parse json in log. How can I config filebeat.

Please help me, thanks a lot.

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

I have just update my question:

This is my log file:

2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 controllers.go:129: [info] RefreshToken | Request Data: [RefreshToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkwOTEyNzMsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xX2EwOTkzZjc3LThhNTAtNDY0MC04NmFiLTBjNTFkOTQwODRlYiIsInVzZXJfaWQiOjF9.r7dNAaFAeW0IMCl5kmbN17CY7wJd98iEwxyoOCC-0B0]
2022-05-02 15:43:12 controllers.go:145: [info] RefreshToken | Response Data: {"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NfdXVpZCI6IklWUl9BVF8xX2MyMjE2ZGE4LWFlMGYtNDU2ZC04NGYzLTAwYTM0MzcwMTJmOSIsImF1dGhvcml6ZWQiOnRydWUsImV4cCI6MTY1Mjc3Njk5MiwidXNlcl9pZCI6MX0.drFFIxiZEWBaar4jWbSX8t7Xxmb7R8UbpVsDCxBITbw","msg":"Success","refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkyNDA5OTIsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xXzM1ZmI4MTg1LWFmOTItNDQ3NC1hMWU1LTE5NmVkZmEwZjRjNiIsInVzZXJfaWQiOjF9.tnjIkckL6OEpAQHtQyT6uCAR9OoSr-GgvZgOYGGEXW0","status":0}
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 controllers.go:129: [info] RefreshToken | Request Data: [RefreshToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkyNDA5OTIsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xXzM1ZmI4MTg1LWFmOTItNDQ3NC1hMWU1LTE5NmVkZmEwZjRjNiIsInVzZXJfaWQiOjF9.tnjIkckL6OEpAQHtQyT6uCAR9OoSr-GgvZgOYGGEXW0]
2022-05-02 15:43:12 controllers.go:145: [info] RefreshToken | Response Data: {"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NfdXVpZCI6IklWUl9BVF8xX2U5MDg0ZjBkLWQzOTQtNDdiYi04NGU0LWEwNmE3NDg4ZDU0ZSIsImF1dGhvcml6ZWQiOnRydWUsImV4cCI6MTY1Mjc3Njk5MiwidXNlcl9pZCI6MX0.yB21voeeKCskTH3nxvwV8pojv3z4r8Bwy4xEtZjMhIw","msg":"Success","refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkyNDA5OTIsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xXzFkM2U1NzIzLTEwZTItNDA3MC1iYmQ3LThmMzMwMTJhNmU2NiIsInVzZXJfaWQiOjF9.NNt2E5kxjJf6gePh4LUZJX27n2h7gqAABMIzjIil6a8","status":0}
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported

This is my filebeat configuaration:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: efk-xpack
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/lib/docker/containers/smartivr-go-backend-after.log
    #   processors:
    #     - decode_json_fields:
    #               fields: ["message"]
    #               process_array: false
    #               max_depth: 2
    #               target: ""
    #               overwrite_keys: true
    #               add_error_key: false
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/lib/docker/containers/smartivr-go-backend-after.log"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          processors:
            - decode_json_fields:
              fields: ["message"]
              process_array: false
              max_depth: 2
              target: ""
              overwrite_keys: true
              add_error_key: false
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/lib/docker/containers/smartivr-go-backend-after.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: efk-xpack
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.10.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch-client.efk-xpack.svc.cluster.local
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /opt/smartivr-storage/smartivr-go-backend
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: efk-xpack
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: efk-xpack
  labels:
    k8s-app: filebeat
---

This is announcement from filebeat:

2022-05-03T18:02:55.020Z	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.10.0
2022-05-03T18:02:55.020Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.10.0' as ILM is enabled.
2022-05-03T18:02:55.020Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://elasticsearch-client.efk-xpack.svc.cluster.local:9200
2022-05-03T18:02:55.021Z	INFO	[publisher]	pipeline/module.go:113	Beat name: callbot-29
2022-05-03T18:02:55.022Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2022-05-03T18:02:55.022Z	INFO	instance/beat.go:455	filebeat start running.
2022-05-03T18:02:55.024Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=29158
2022-05-03T18:02:55.315Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=42444
2022-05-03T18:02:55.315Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 60
2022-05-03T18:02:55.316Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2022-05-03T18:02:55.316Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2022-05-03T18:02:55.316Z	WARN	[cfgwarn]	kubernetes/config.go:84	DEPRECATED: `host` will be deprecated, use `node` instead Will be removed in version: 8.0
2022-05-03T18:02:55.317Z	WARN	[cfgwarn]	kubernetes/config.go:84	DEPRECATED: `host` will be deprecated, use `node` instead Will be removed in version: 8.0
2022-05-03T18:02:55.317Z	INFO	[autodiscover.pod]	kubernetes/util.go:99	kubernetes: Using node 103.141.141.29 provided in the config
2022-05-03T18:02:55.317Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager
2022-05-03T18:02:55.419Z	INFO	log/input.go:157	Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.420Z	INFO	log/input.go:157	Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.420Z	INFO	log/input.go:157	Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:302	Harvester started for file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	INFO	log/input.go:157	Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z	INFO	log/harvester.go:335	Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"

How can I solve this ? Please.
And I want to parse json in log. How can I config filebeat.

Please help me, thanks a lot.

1 Like

Hi @Hoa_Nguy_n_Van,

How /var/lib/docker/containers/smartivr-go-backend-after.log log file is generated?

two expected log formats are docker, or cri: CRI log format example, docker log format example.

Why are you using decode_json_fields processor, format of the log file is not a json, also instead of processor can be used json.* configuration for container input.