Filebeat versions from 7.0 - 7.8 fail to create alias field mappings for majority of modules

Summary: When running Filebeat's 7.8.0 setup command the field mappings for the majority of modules which define mappings of type alias never get created. The alias mappings are missing from export template output as well. Code review shows this is affecting only aliases declared with migration: true in fields.yaml.

Details: This was initially discovered while attempting to perform a first-time setup using Filebeat package v7.8.1 installed on Ubuntu 18.04 LTS from Elastic's Debian repository at https://artifacts.elastic.co/packages/7.x/apt (suite: stable/main, components: amd64, Packages) and Nginx module enabled with "filebeat modules enable nginx" using a new Elasticsearch Service on Elastic Cloud deployment (Elasticsearch 7.8) as the configured output of Filebeat. None of the alias fields listed in the Filebeat Reference 7.8, Exported fields, Nginx fields were present in the created index template.

A closer investigation of the Filebeat log output with logging level set to debug as well as the output of filebeat setup template reveals that it's not only Nginx module that is affected. The only modules that have alias mappings present are traefik and suricata with all of the others (e.g. Apache, Beat, Docker, Elasticsearch, Kafka and Redis) affected by this.

Temporary workaround: After some digging around both open and closed Beats Github issues I have come across pull request: Do not configure aliases in unsupported Elasticsearch versions #9992 which introduced a check in "libbeat/template/processor.go" to only configure aliases in Elasticsearch versions 6.4 and above. Commenting it out did resolve the issue, but commenting the conditional check "if !p.Migration && f.MigrationAlias" that followed it (commented as "In case migration is disabled and it's a migration alias, field is not created") actually did:

func (p *Processor) alias(f *mapping.Field) common.MapStr {
        // Aliases were introduced in Elasticsearch 6.4, ignore if unsupported
        if p.EsVersion.LessThan(common.MustNewVersion("6.4.0")) {
                return nil
        }

        // In case migration is disabled and it's a migration alias, field is not created
        //if !p.Migration && f.MigrationAlias {
        //      return nil
        //}
        properties := getDefaultProperties(f)
        properties["type"] = "alias"
        properties["path"] = f.AliasPath

        return properties
}

Following this modification the missing aliases are now present when running filebeat setup and also in the output of filebeat export template.

Here's part of the filebeat export template output for the Nginx module before the modification:

{
  "nginx": {
    "properties": {
      "access": {
        "properties": {
          "geoip": {
            "properties": {}
          },
          "user_agent": {
            "properties": {}
          }
        }
      },
      "ingress_controller": {
        "properties": {
          "upstream": {
            "properties": {
              "alternative_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "port": {
                "type": "long"
              },
              "response": {
                "properties": {
                  "status_code": {
                    "type": "long"
                  },
                  "length": {
                    "type": "long"
                  },
                  "time": {
                    "type": "double"
                  }
                }
              },
              "ip": {
                "type": "ip"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "geoip": {
            "properties": {}
          },
          "http": {
            "properties": {
              "request": {
                "properties": {
                  "length": {
                    "type": "long"
                  },
                  "time": {
                    "type": "double"
                  },
                  "id": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  }
                }
              }
            }
          },
          "user_agent": {
            "properties": {}
          }
        }
      },
      "error": {
        "properties": {
          "connection_id": {
            "type": "long"
          }
        }
      }
    }
  }
}

And after the modification:

{
  "nginx": {
    "properties": {
      "access": {
        "properties": {
          "referrer": {
            "path": "http.request.referrer",
            "type": "alias"
          },
          "response_code": {
            "path": "http.response.status_code",
            "type": "alias"
          },
          "agent": {
            "path": "user_agent.original",
            "type": "alias"
          },
          "geoip": {
            "properties": {
              "continent_name": {
                "path": "source.geo.continent_name",
                "type": "alias"
              },
              "region_iso_code": {
                "path": "source.geo.region_iso_code",
                "type": "alias"
              },
              "city_name": {
                "path": "source.geo.city_name",
                "type": "alias"
              },
              "country_iso_code": {
                "path": "source.geo.country_iso_code",
                "type": "alias"
              },
              "location": {
                "path": "source.geo.location",
                "type": "alias"
              },
              "region_name": {
                "path": "source.geo.region_name",
                "type": "alias"
              }
            }
          },
          "method": {
            "path": "http.request.method",
            "type": "alias"
          },
          "user_name": {
            "path": "user.name",
            "type": "alias"
          },
          "http_version": {
            "path": "http.version",
            "type": "alias"
          },
          "body_sent": {
            "properties": {
              "bytes": {
                "path": "http.response.body.bytes",
                "type": "alias"
              }
            }
          },
          "url": {
            "path": "url.original",
            "type": "alias"
          },
          "user_agent": {
            "properties": {
              "original": {
                "path": "user_agent.original",
                "type": "alias"
              },
              "os": {
                "path": "user_agent.os.full_name",
                "type": "alias"
              },
              "name": {
                "path": "user_agent.name",
                "type": "alias"
              },
              "os_name": {
                "path": "user_agent.os.name",
                "type": "alias"
              },
              "device": {
                "path": "user_agent.device.name",
                "type": "alias"
              }
            }
          }
        }
      },
      "ingress_controller": {
        "properties": {
          "referrer": {
            "path": "http.request.referrer",
            "type": "alias"
          },
          "agent": {
            "path": "user_agent.original",
            "type": "alias"
          },
          "response_code": {
            "path": "http.response.status_code",
            "type": "alias"
          },
          "upstream": {
            "properties": {
              "alternative_name": {
                "ignore_above": 1024,
                "type": "keyword"
              },
              "port": {
                "type": "long"
              },
              "response": {
                "properties": {
                  "status_code": {
                    "type": "long"
                  },
                  "length": {
                    "type": "long"
                  },
                  "time": {
                    "type": "double"
                  }
                }
              },
              "ip": {
                "type": "ip"
              },
              "name": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "geoip": {
            "properties": {
              "region_iso_code": {
                "path": "source.geo.region_iso_code",
                "type": "alias"
              },
              "continent_name": {
                "path": "source.geo.continent_name",
                "type": "alias"
              },
              "city_name": {
                "path": "source.geo.city_name",
                "type": "alias"
              },
              "country_iso_code": {
                "path": "source.geo.country_iso_code",
                "type": "alias"
              },
              "location": {
                "path": "source.geo.location",
                "type": "alias"
              },
              "region_name": {
                "path": "source.geo.region_name",
                "type": "alias"
              }
            }
          },
          "method": {
            "path": "http.request.method",
            "type": "alias"
          },
          "user_name": {
            "path": "user.name",
            "type": "alias"
          },
          "http": {
            "properties": {
              "request": {
                "properties": {
                  "length": {
                    "type": "long"
                  },
                  "id": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "time": {
                    "type": "double"
                  }
                }
              }
            }
          },
          "http_version": {
            "path": "http.version",
            "type": "alias"
          },
          "body_sent": {
            "properties": {
              "bytes": {
                "path": "http.response.body.bytes",
                "type": "alias"
              }
            }
          },
          "user_agent": {
            "properties": {
              "original": {
                "path": "user_agent.original",
                "type": "alias"
              },
              "os": {
                "path": "user_agent.os.full_name",
                "type": "alias"
              },
              "name": {
                "path": "user_agent.name",
                "type": "alias"
              },
              "os_name": {
                "path": "user_agent.os.name",
                "type": "alias"
              },
              "device": {
                "path": "user_agent.device.name",
                "type": "alias"
              }
            }
          },
          "url": {
            "path": "url.original",
            "type": "alias"
          }
        }
      },
      "error": {
        "properties": {
          "connection_id": {
            "type": "long"
          },
          "level": {
            "path": "log.level",
            "type": "alias"
          },
          "pid": {
            "path": "process.pid",
            "type": "alias"
          },
          "message": {
            "path": "message",
            "type": "alias"
          },
          "tid": {
            "path": "process.thread.id",
            "type": "alias"
          }
        }
      }
    }
  }
}

I went back through all Filebeat releases, downloading and checking output of filebeat export tempalte for versions 7.0, 7.1 etc. to version 7.7 and all exhibited the same behavior. The last version where this was not the case was version 6.8 where fields.yaml did not specify the types of the fields as aliases yet.

Other users seem to have encountered this, e.g. user @prophoto was searching for Apache field names but failed to find any in Kibana's Discover, see the thread 7.8 doesn’t parse apache access logs.

1 Like

Concluding thoughts: While the missing aliases do not affect the modules' ingest pipelines as writing to alias fields is unsupported (see the Unsupported APIs section of Elasticsearch's Alias data type documentation) the user experience in Discover does suffer as the user is forced to look up the aliases' target paths in the documentation. Instead of the aliases serving as a convenient mechanism for generating a module-specific hierarchical namespace of fields all accessible via a common prefix (e.g. nginx.*) the user is thus forced to memorize the fields scattered all over across the ECS field sets.

The fields.yaml does declare the missing alias attributes with "migration: true" so the processor.go's func (p *Processor) alias(f *mapping.Field) behaves as instructed - it skips creating them. Can anybody offer some comments on why this is though, i.e. why the aliases would only be created as part of a migration? I have failed to notice any deprecation warnings in the Filebeat modules documentation that would mention the aliases are getting deprecated. A bug or a feature?

Filebeat documentation should be updated accordingly if the module aliases are in fact getting deprecated.

Here might be a helpful doc for you: https://www.elastic.co/guide/en/beats/libbeat/current/upgrading-6-to-7.html

@mtojek Thank you for providing the link! Breaking changes in 7.0 is the link that answers my questions/concerns regarding the aliases and contains the full list of fields that were renamed with the switch to ECS in 7.0.

Nevertheless I would highly recommend the Filebeat documentation to be updated as the Exported fields chapter is misleading new users into believing the aliases are supposed to be present. The top Exported fields page should explicitly mention that the aliases will be created during setup only in case "migration.6_to_7.enabled: true" is configured. Additionally each module page should have a deprecation warning for the listed aliases.

Should a GitHub issue be opened requesting this change/improvement of the documentation?

@mtojek Another note: the change of aliases not being created is now leaving nested structures for the Filebeat modules with "properties" JSON objects containing no attributes, e.g.:

{
  "nginx": {
    "properties": {
      "access": {
        "properties": {
          "geoip": {
            "properties": {}
          },
          "user_agent": {
            "properties": {}
          }
        }
      }
    }
  }
}

These entries serve no purpose and only present clutter. Any paths with edge/leaf nodes not containing actual field definitions should be pruned from the tree structure.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.