Hello,
I have a problem in creating index json in filebeat. It can not parse log file.
This is my log file:
This is my filebeat config:
This is announcement from filebeat:
How can I solve this ? Please.
And I want to parse json in log. How can I config filebeat.
Please help me, thanks a lot.
Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them 
I have just update my question:
This is my log file:
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 controllers.go:129: [info] RefreshToken | Request Data: [RefreshToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkwOTEyNzMsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xX2EwOTkzZjc3LThhNTAtNDY0MC04NmFiLTBjNTFkOTQwODRlYiIsInVzZXJfaWQiOjF9.r7dNAaFAeW0IMCl5kmbN17CY7wJd98iEwxyoOCC-0B0]
2022-05-02 15:43:12 controllers.go:145: [info] RefreshToken | Response Data: {"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NfdXVpZCI6IklWUl9BVF8xX2MyMjE2ZGE4LWFlMGYtNDU2ZC04NGYzLTAwYTM0MzcwMTJmOSIsImF1dGhvcml6ZWQiOnRydWUsImV4cCI6MTY1Mjc3Njk5MiwidXNlcl9pZCI6MX0.drFFIxiZEWBaar4jWbSX8t7Xxmb7R8UbpVsDCxBITbw","msg":"Success","refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkyNDA5OTIsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xXzM1ZmI4MTg1LWFmOTItNDQ3NC1hMWU1LTE5NmVkZmEwZjRjNiIsInVzZXJfaWQiOjF9.tnjIkckL6OEpAQHtQyT6uCAR9OoSr-GgvZgOYGGEXW0","status":0}
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 controllers.go:129: [info] RefreshToken | Request Data: [RefreshToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkyNDA5OTIsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xXzM1ZmI4MTg1LWFmOTItNDQ3NC1hMWU1LTE5NmVkZmEwZjRjNiIsInVzZXJfaWQiOjF9.tnjIkckL6OEpAQHtQyT6uCAR9OoSr-GgvZgOYGGEXW0]
2022-05-02 15:43:12 controllers.go:145: [info] RefreshToken | Response Data: {"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NfdXVpZCI6IklWUl9BVF8xX2U5MDg0ZjBkLWQzOTQtNDdiYi04NGU0LWEwNmE3NDg4ZDU0ZSIsImF1dGhvcml6ZWQiOnRydWUsImV4cCI6MTY1Mjc3Njk5MiwidXNlcl9pZCI6MX0.yB21voeeKCskTH3nxvwV8pojv3z4r8Bwy4xEtZjMhIw","msg":"Success","refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3MjkyNDA5OTIsInJlZnJlc2hfdXVpZCI6IklWUl9SVF8xXzFkM2U1NzIzLTEwZTItNDA3MC1iYmQ3LThmMzMwMTJhNmU2NiIsInVzZXJfaWQiOjF9.NNt2E5kxjJf6gePh4LUZJX27n2h7gqAABMIzjIil6a8","status":0}
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
2022-05-02 15:43:12 role_check.go:17: [error] middleware.CheckRole: sql: Scan error on column index 0, name "organization_id": converting NULL to int is unsupported
This is my filebeat configuaration:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: efk-xpack
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
# filebeat.inputs:
# - type: container
# paths:
# - /var/lib/docker/containers/smartivr-go-backend-after.log
# processors:
# - decode_json_fields:
# fields: ["message"]
# process_array: false
# max_depth: 2
# target: ""
# overwrite_keys: true
# add_error_key: false
# - add_kubernetes_metadata:
# host: ${NODE_NAME}
# matchers:
# - logs_path:
# logs_path: "/var/lib/docker/containers/smartivr-go-backend-after.log"
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
filebeat.autodiscover:
providers:
- type: kubernetes
host: ${NODE_NAME}
processors:
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 2
target: ""
overwrite_keys: true
add_error_key: false
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/lib/docker/containers/smartivr-go-backend-after.log
processors:
- add_cloud_metadata:
- add_host_metadata:
cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: efk-xpack
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.10.0
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch-client.efk-xpack.svc.cluster.local
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /opt/smartivr-storage/smartivr-go-backend
- name: varlog
hostPath:
path: /var/log
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: efk-xpack
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: efk-xpack
labels:
k8s-app: filebeat
---
This is announcement from filebeat:
2022-05-03T18:02:55.020Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.0
2022-05-03T18:02:55.020Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.0' as ILM is enabled.
2022-05-03T18:02:55.020Z INFO eslegclient/connection.go:99 elasticsearch url: http://elasticsearch-client.efk-xpack.svc.cluster.local:9200
2022-05-03T18:02:55.021Z INFO [publisher] pipeline/module.go:113 Beat name: callbot-29
2022-05-03T18:02:55.022Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2022-05-03T18:02:55.022Z INFO instance/beat.go:455 filebeat start running.
2022-05-03T18:02:55.024Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=29158
2022-05-03T18:02:55.315Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=42444
2022-05-03T18:02:55.315Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 60
2022-05-03T18:02:55.316Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 0
2022-05-03T18:02:55.316Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 0
2022-05-03T18:02:55.316Z WARN [cfgwarn] kubernetes/config.go:84 DEPRECATED: `host` will be deprecated, use `node` instead Will be removed in version: 8.0
2022-05-03T18:02:55.317Z WARN [cfgwarn] kubernetes/config.go:84 DEPRECATED: `host` will be deprecated, use `node` instead Will be removed in version: 8.0
2022-05-03T18:02:55.317Z INFO [autodiscover.pod] kubernetes/util.go:99 kubernetes: Using node 103.141.141.29 provided in the config
2022-05-03T18:02:55.317Z INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
2022-05-03T18:02:55.419Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.420Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.420Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.421Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/smartivr-go-backend-after.log]
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
2022-05-03T18:02:55.421Z INFO log/harvester.go:335 Skipping unparsable line in file: /var/lib/docker/containers/smartivr-go-backend-after.log
2022-05-03T18:02:55.421Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2022-05-02" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "" as "T"
How can I solve this ? Please.
And I want to parse json in log. How can I config filebeat.
Please help me, thanks a lot.
Hi @Hoa_Nguy_n_Van,
How /var/lib/docker/containers/smartivr-go-backend-after.log log file is generated?
two expected log formats are docker, or cri: CRI log format example, docker log format example.
Why are you using decode_json_fields processor, format of the log file is not a json, also instead of processor can be used json.* configuration for container input.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.