Using values from an array as field names to contain values from another array

Elastic Stack Elasticsearch
1

I am ingesting event logs from Office 365. Some of these documents contain objects that look like this:

"DeviceProperties": [
          {
            "Value": "Windows10",
            "Name": "OS"
          },
          {
            "Value": "Edge",
            "Name": "BrowserType"
          },
          {
            "Value": "some-uuid-here",
            "Name": "SessionId"
          }
        ],

What I am trying to do is map or (if required) modify this so that queries like

o365.audit.DeviceProperties.SessionId:"some-uuid-here"
o365.audit.DeviceProperties.OS:"Windows10"
o365.audit.DeviceProperties.BrowserType:"Edge"

work the way I'd (naively) expect them to. Neither nested or flattened field types seem to handle this correctly by default. Is there some way to associate the "Name" property as a subfield name with the corresponding "Value" in each object?

2

Hi @ccweirsw

I believe that using flattend or nested you would have to combine each pair of properties in your query.

     {
          "term": {
            "DeviceProperties.Name": {
              "value": "OS"
            }
          }
        },
        {
          "term": {
            "DeviceProperties.Value": {
              "value": "Windows10"
            }
          }
        }

Another option that is close to the way you want to search would be indexing as below:

{
  "DeviceProperties": {
    "OS": "Windows10",
    "BrowserType": "Edge",
    "SessionId": "some-uuid-here"
  }
}
3

Are you saying that modifying the source document to look more like that is what would be needed? Any suggestions on an efficient way to do that at ingest time?

Thanks

4

Yes, because the way it is indexed you will not be able to perform the queries, confirm below:

How do you ingest this data?

5

Logs are ingested from Filebeat.

I think I have a solution to this that uses pipeline processors that I'll post when I've tested it properly. Your comment helped me realize I had to modify the source if I was going to get the behavior I wanted.

1 Like
6 
{
    "foreach": {
      "field": "o365.audit.DeviceProperties",
      "processor": {
        "set": {
          "field": "_temp_.DeviceProperties.{{{_ingest._value.Name}}}",
          "value": "{{{_ingest._value.Value}}}"
        }
      },
      "tag": "deviceproperties-rewrite",
      "ignore_failure": true,
      "description": "Copies key:value pairs out of o365.audit.DeviceProperties to a temp field"
    }
  },
  {
    "remove": {
      "field": "o365.audit.DeviceProperties",
      "ignore_missing": true,
      "tag": "deviceproperties-remove-original"
    }
  },
  {
    "rename": {
      "field": "_temp_.DeviceProperties",
      "target_field": "o365.audit.DeviceProperties",
      "ignore_missing": true,
      "tag": "deviceproperties-rename",
      "description": "Copy the new key:value structure back to the original field"
    }
  },

This is an excerpt of the ingest pipeline that solved the issue. Just uses a foreach processor to copy all the field names and values I want to a temp field, remove the original one, and then copy the data back in the format I want to use.