Logs parsing error post upgrading filebeat from 5.x to 7.x

We recently updated the filebeat to 7.9.0 for ECS Clusters but after updating it stopped parsing the logs and sending file path in the log field.

sample log:

log": {
      "file": {
        "path": "/var/lib/docker/containers/<container_id>/<cluster_name>@<container_name>@<tag_name>@0@<service_name>@latest.nlog"

Sample logs before updating the filebeat:

"log": "<actual_log>"
"source": "/var/lib/docker/containers/<container_id>/<cluster_name>@<container_name>@<tag_nmae>@0@<service_name>@latest.nlog"

Filebeat Config:

- type: log
  enabled: true
    - /var/lib/docker/containers/*/*.nlog
  json.keys_under_root: true
  json.message_key: log
  json.add_error_key: true
  json.overwrite_keys: true
  symlinks: true
  fields_under_root: true
    level: debug
    type: docker_json
    category: <index_name>

logstash mapping:

before (with 5.x)

    match => {
        "source" => "%{GREEDYDATA}\/%{GREEDYDATA:ecs_cluster_name}@%{GREEDYDATA:ecs_container_name}@%{GREEDYDATA:ecs_task_definition_family}@%{GREEDYDATA:ecs_task_definition_version}@%{GREEDYDATA:ecs_image_basename}@%{GREEDYDATA:ecs_image_tag}.nlog"

With 7.9:

    match => {
        "[log][file][path]" => "%{GREEDYDATA}\/%{GREEDYDATA:ecs_cluster_name}@%{GREEDYDATA:ecs_container_name}@%{GREEDYDATA:ecs_task_definition_family}@%{GREEDYDATA:ecs_task_definition_version}@%{GREEDYDATA:ecs_image_basename}@%{GREEDYDATA:ecs_image_tag}.nlog"

So instead of file content, there is path field. So we suspect, its something to do with source and log.file.path.

any help would be appreciated.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.