Conflicts in Azure Fleet Integrations

[PhilA (Phil)]

Hi

I think there is an issue with the current Azure logs integration for Fleet (Azure logs integration v0.8.5). I am working with this to try and pull in various logs from Azure and have noticed field conflicts in my logs-* index pattern. These are all Elastic generated indicies.

Steps taken to investigate so far:

The index patterns page tells me that there are 2 conflicts; client.ip and error.message.

Using dev tools, I ran
GET /logs-*/_mapping/field/client.ip

This returned lots but picking out 2 key bits here, note that 2 indicies, both Azure default ones have client.ip field listed and one shows type IP and the other shows type keyword. It seems that the following have client.ip set to keyword:

• platformlogs
• auditlogs

But the following are correct and have it set as type IP:

• signinlogs
• activitylogs

  ".ds-logs-azure.platformlogs-default-2021.09.02-000001" : {
    "mappings" : {
      "client.ip" : {
        "full_name" : "client.ip",
        "mapping" : {
          "ip" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  },
  ".ds-logs-elastic_agent.filebeat-default-2021.06.15-000008" : {
    "mappings" : { }
  },
  ".ds-logs-elastic_agent.metricbeat-default-2021.08.30-000009" : {
    "mappings" : { }
  },
  ".ds-logs-azure.activitylogs-default-2021.09.02-000001" : {
    "mappings" : {
      "client.ip" : {
        "full_name" : "client.ip",
        "mapping" : {
          "ip" : {
            "type" : "ip"
          }

There are also conflicts on the error.message field which sometimes shows as type text and other times keyword. I can get more details on this if needed but just wanted to start by verifying if I am correct and this is a bug.

Thanks

[ruflin]

I have not digged into in details but I assume these fields are missing in the package: https://github.com/elastic/integrations/tree/master/packages/azure/data_stream/platformlogs/fields

@MarianaD To not have to add all ECS fields with *.ip, maybe we can add a dynamic mapping that matches?

[MarianaD]

@PhilA, thanks for brining this up, we need to check how those fields are defined in the ecs schema and make sure they are consistent over the integration. Feel free to create an issue on this, else I will create on based on this.

[PhilA (Phil)]

Thanks @MarianaD, I have raised an issue:

https://github.com/elastic/integrations/issues/1594

Hopefully that is ok, I'm fairly new to Github