Logs parsing error post upgrading filebeat from 5.x to 7.x

Elastic Stack Beats
,
1

We recently updated the filebeat to 7.9.0 for ECS Clusters but after updating it stopped parsing the logs and sending file path in the log field.

sample log:

log": {
      "file": {
        "path": "/var/lib/docker/containers/<container_id>/<cluster_name>@<container_name>@<tag_name>@0@<service_name>@latest.nlog"
      },

Sample logs before updating the filebeat:

{
....
....
"log": "<actual_log>"
"source": "/var/lib/docker/containers/<container_id>/<cluster_name>@<container_name>@<tag_nmae>@0@<service_name>@latest.nlog"
....
....
}

Filebeat Config:

 filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/lib/docker/containers/*/*.nlog
  json.keys_under_root: true
  json.message_key: log
  json.add_error_key: true
  json.overwrite_keys: true
  symlinks: true
  fields_under_root: true
  fields:
    level: debug
    type: docker_json
    category: <index_name>

logstash mapping:

before (with 5.x)

    match => {
        "source" => "%{GREEDYDATA}\/%{GREEDYDATA:ecs_cluster_name}@%{GREEDYDATA:ecs_container_name}@%{GREEDYDATA:ecs_task_definition_family}@%{GREEDYDATA:ecs_task_definition_version}@%{GREEDYDATA:ecs_image_basename}@%{GREEDYDATA:ecs_image_tag}.nlog"
      }
    }

With 7.9:

    match => {
        "[log][file][path]" => "%{GREEDYDATA}\/%{GREEDYDATA:ecs_cluster_name}@%{GREEDYDATA:ecs_container_name}@%{GREEDYDATA:ecs_task_definition_family}@%{GREEDYDATA:ecs_task_definition_version}@%{GREEDYDATA:ecs_image_basename}@%{GREEDYDATA:ecs_image_tag}.nlog"
      }
    }

So instead of file content, there is path field. So we suspect, its something to do with source and log.file.path.

any help would be appreciated.

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.