Error logging container json files with filebeat

Elastic Stack Beats
,
1

Hello,

I am struggling for days on parsing a simple airflow json file from a k8s container

Log format:
{"asctime": "2021-12-02 17:04:34,450", "filename": "standard_task_runner.py", "lineno": 77, "levelname": "INFO", "message": "Job 3735: Subtask templated", "offset": 1638464674450166272, "dag_id": "tutorial_git", "task_id": "templated", "execution_date": "2021_12_02T16_59_29_088697", "try_number": "1", "log_id": "tutorial_git_templated_2021_12_02T16_59_29_088697_1"}

filebeat configuration:

filebeat:
  config:
    modules:
      path: /usr/share/filebeat/modules.d/*.yml
      reload:
        enabled: true

  inputs:
    - type: container
      combine_partial: true
      paths:
        - /opt/airflow/logs/*/*/*/*.log*

      json.keys_under_root: true
      json.add_error_key: true

error:

2021-12-03T11:57:14.064Z	ERROR	[reader_docker_json]	readjson/docker_json.go:204	Parse line error: parsing docker timestamp: parsing time "" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "2006"
2021-12-03T11:57:14.064Z	INFO	log/harvester.go:326	Skipping unparsable line in file: /opt/airflow/logs/tutorial_git/templated/2021-12-02T16:59:29.088697+00:00/1.log

I tried decode json fields, combine lines, timestamp processor... dont know what is wrong

Regards
Guillaume

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.