Elastic 7.9.1 - Security (SIEM) - Your visualization has error(s) - [illegal_argument_exception]

Hi,

We have a brand new cluster install to test 7.9.1 and the security features in particular.

When navigating to the Overview page we get the following error.

[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.module] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

So far I have only ingested 3 sources via Winlogbeat, Filebeat and Logstash. I have also tried to limit the indices to only Winlogbeat but still get the same result.

Anything I can try to troubleshoot?

Hi @Derick_Jansen, thanks for checking out the Security app!

The Overview page includes an External alert trend that performs an aggregation on the event.module field.

The Elastic Common Schema (ECS) defines the event.module field as a keyword field, as documented here.

Given the following information:

it appears that Elasticsearch inferred a different (text) mapping for the event.module field when events were ingested via Logstash. In contrast to Logstash, events ingested by Winlogbeat and Filebeat automatically conform to the Elastic Common Schema. If (by luck) you happened to ingest data via Winlogbeat or Filebeat first, Elasticsearch would have automatically used the correct mapping for event.module, (keyword instead of text).

You can't change the mapping of an existing field because changing an existing field could invalidate data that's already indexed. The change the mapping of an existing field documentation states that if you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index. That said, since this is a new cluster for testing, and we don't know what other fields may have been mapped with unexpected types, the simplest solution would be to drop the existing indexes, and re-ingest the data (first) with Winlogbeat and Filebeat to ensure the mapping event.module (and other fields) are mapped as documented in the ECS Field Reference.

Hi @Andrew_G

Thanks a lot for your reply. I initially started with Filebeat using the Fortinet module so ECS should have been in place.

I tried your suggestion anyway. I stopped all the beats and Logstash, dropped the indexes and templates and started only Winlogbeat.

Unfortunately, I am still getting the same error as before.

Thanks again for your help.

Let's make sure the mapping for event.module is keyword for all the indicies in your environment.

The default indicies used by the Security app

By default, the Security app displays data from the following Elasticsearch indicies, as specified in the securitySolution:defaultIndex Kibana Advanced setting:

apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, packetbeat-*, winlogbeat-*

Note: You can view the above Kibana Advanced setting via Stack Management > Kibana > Advanced Settings.

About the _mapping API examples in this reply

The next section makes use of the _mapping API to view the mappings for indicies that match the index patterns specified in the securitySolution:defaultIndex Kibana Advanced setting.

To provide the example output in this reply, I spun up a fresh instance of 7.9.2 in Elastic Cloud, and then ingested data from Auditbeat on my Mac to populate the Security app with data:

./auditbeat setup
./auditbeat -e

Using the _mapping API

Let's use the mapping API to view the mappings for each of the indexes specified in the securitySolution:defaultIndex Kibana Advanced setting.

To get started, navigate to Management > Dev Tools.

In the next sections, we'll execute commands in the Dev Tools.

apm-*-transaction*

Execute the following query, for the first index specified securitySolution:defaultIndex Kibana Advanced setting, apm-*-transaction*:

GET /apm-*-transaction*/_mapping/field/event.module

The output for the above command in the environment described above is:

{
  "apm-7.9.2-transaction-000001" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

The output above shows the mapping for event.moulde in the apm-7.9.2-transaction-000001 index.

auditbeat-*

Execute the following query:

GET /auditbeat-*/_mapping/field/event.module

The output for the above command in the environment described above is:

{
  "auditbeat-7.9.2-2020.09.29-000001" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

endgame-*

Execute the following query:

GET /endgame-*/_mapping/field/event.module

The output for the above command in the environment described above is:

{ }

filebeat-*

Execute the following query:

GET /filebeat-*/_mapping/field/event.module

The output for the above command in the environment described above is:

{ }

Since you ingested data via Filebeat in your environment, I would expect to see one or more indicies when the GET command above is run.

logs-*

Execute the following query:

GET /logs-*/_mapping/field/event.module

The output for the above command in the environment described above is:

{ }

`packetbeat-*

Execute the following query:

GET /packetbeat-*/_mapping/field/event.module

The output for the above command in the environment described above is:

{ }

winlogbeat-*

Execute the following query:

GET /winlogbeat-*/_mapping/field/event.module

The output for the above command in the environment described above is:

{ }

Since you ingested data via Winlogbeat in your environment, you may see one or more indicies when the GET command above is run.

Mappings for Detections indicies

If you activated any detection rules, detection alerts will also be written indexes that match the .siem-signals-default index pattern. To view the event.module mappings for Detections indicies, execute the following query:

GET /.siem-signals-default/_mapping/field/event.module

The output for the above command in the environment described above is:

{
  ".siem-signals-default-000001" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

The output above shows detection alerts were enabled in the sample environment described above, and indexed into the .siem-signals-default-000001.

@Derick_Jansen, would you be willing to validate the mappings for event.module via the steps above?

Hi @Andrew_G

At this point, I only have a Winlogbeat index.

securitySolution:defaultIndex is also set to only winlogbeat-*

GET /winlogbeat-*/_mapping/field/event.module
    {
      "winlogbeat-7.9.2" : {
        "mappings" : {
          "event.module" : {
            "full_name" : "event.module",
            "mapping" : {
              "module" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          }
        }
      }
    }
GET /.siem-signals-default/_mapping/field/event.module   
{
  ".siem-signals-default-000001" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

Thanks for sharing the output of the GET /winlogbeat-*/_mapping/field/event.module command @Derick_Jansen!

Summary

I'll explain the problem in a bit more detail below, but to resolve the error, would you be willing to perform the following steps?

  1. Load the Winlogbeat index template manually

  2. Re-index the old Winlogbeat data and delete the old indexes that were created before step 1. (Since this cluster was created just for testing, you can also just delete the old indexes.)

Details

Winlogbeat creates the index template and mappings when the winlogbeat setup command is run:

winlogbeat setup

It looks like the Winlogbeat index templates in your environment were created when the Logstash data was ingested, as opposed to being setup by Winlogbeat.

How did we determine this? The output of

GET /.siem-signals-default/_mapping/field/event.module   

in your environment:

{
  ".siem-signals-default-000001" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

has the expected mapping for event.module, but the output of

GET /winlogbeat-*/_mapping/field/event.module

in your environment:

{
      "winlogbeat-7.9.2" : {
        "mappings" : {
          "event.module" : {
            "full_name" : "event.module",
            "mapping" : {
              "module" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }
            }
          }
        }
      }
    }

contains an unexpected

"type" : "text",

I would expect the mapping for event.module in your WInlogbeat index to have the same shape as the mapping reported by running GET /.siem-signals-default/_mapping/field/event.module.

The different shape suggests that loading the index template manually and reindexing+dropping the old indexes will resolve the error.

Would you be willing to try the steps above to see if that resolves the error?

Hi @Andrew_G

I am seeing some strange behaviour. It seems like the template is being ignored.

I will step through what I have done. (I am testing only with Winlogbeat to make it easier)

DELETE /_template/winlogbeat-*
DELETE /winlogbeat-*

GET _template/winlogbeat-7.9.2/
{}
GET /winlogbeat-*
{}

Everything cleared out
Run setup to create the template

 GET _template/winlogbeat-7.9.2/  
    ...
    "event" : {
      "properties" : {
        "severity" : {
          "type" : "long"
        },
        "code" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "original" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "risk_score" : {
          "type" : "float"
        },
        "timezone" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "created" : {
          "type" : "date"
        },
        "kind" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "module" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        ...

As we can see it created the template with event.module as a keyword.

Index is still empty at this point.

GET /winlogbeat-*/_mapping/field/event.module
{ }

Ingest some data ...

{
  "winlogbeat-7.9.2" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    }
  }
}

Problem is still there.

In short the template is correct but the index is created with the incorrect field type.

Any ideas? Thanks again I really appreciate all the help.

Cheers!

Thanks for your detailed reply @Derick_Jansen, that's very helpful!

To determine whether or not a global template might be effecting all string fields, would you be willing to share the output of the following command?

GET /_index_template/*

Here you go @Andrew_G

GET /_index_template/*
{
  "index_templates" : [
    {
      "name" : "metrics",
      "index_template" : {
        "index_patterns" : [
          "metrics-*-*"
        ],
        "composed_of" : [
          "metrics-mappings",
          "metrics-settings"
        ],
        "priority" : 100,
        "version" : 0,
        "_meta" : {
          "description" : "default metrics template installed by x-pack",
          "managed" : true
        },
        "data_stream" : { }
      }
    },
    {
      "name" : "logs",
      "index_template" : {
        "index_patterns" : [
          "logs-*-*"
        ],
        "composed_of" : [
          "logs-mappings",
          "logs-settings"
        ],
        "priority" : 100,
        "version" : 0,
        "_meta" : {
          "description" : "default logs template installed by x-pack",
          "managed" : true
        },
        "data_stream" : { }
      }
    }
  ]
}

Thanks for sharing the output of GET /_index_template/* @Derick_Jansen! I referred your post to the Elasticsearch team, because I'm not seeing anything in the output that would explain the unexpected mapping.

Heya @Derick_Jansen, @Andrew_G pinged me to help out :slight_smile:

This also happens with metricbeat, etc right, not just winlogbeat? I don't have a windows machine to test with, but if you see the same behavior with e.g. metricbeat I'll spin up an environment to see if I can reproduce it locally.

After you delete the templates and indices, then re-run setup, could you get the mapping before data is ingested?

E.g. just a simple GET /winlogbeat-7.9.2/_mapping call, prior to ingesting data. I'm curious to see if A) there is a dynamic template being added in somewhere that is modifying the mapping, or B) if the mapping is correct post-setup but something about the data triggers a re-mapping (e.g. using dynamic mapping).

Thanks! Sorry this has caused such a headache so far :slight_smile:

Hi @polyfractal

Thanks a lot for the ongoing help I really do appreciate it!

After setup but before data ingestion ...

GET /winlogbeat-*
{ }

Only the template is created as mentioned above.

GET _template/winlogbeat-7.9.2/
    ...
    "event" : {
      "properties" : {
        "severity" : {
          "type" : "long"
        },
        "code" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "original" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "risk_score" : {
          "type" : "float"
        },
        "timezone" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "created" : {
          "type" : "date"
        },
        "kind" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        "module" : {
          "ignore_above" : 1024,
          "type" : "keyword"
        },
        ...

Output of setup

winlogbeat.exe setup -e -c C:\ProgramData\Elastic\Beats\winlogbeat\winlogbeat.yml
2020-10-06T10:40:36.756+1300    INFO    instance/beat.go:640    Home path: [C:\Program Files\Elastic\Beats\7.9.2\winlogbeat] Config path: [C:\Program Files\Elastic\Beats\7.9.2\winlogbeat] Data path: [C:\Program Files\Elastic\Beats\7.9.2\winlogbeat\data] Logs path: [C:\Program Files\Elastic\Beats\7.9.2\winlogbeat\logs]
2020-10-06T10:40:36.758+1300    INFO    instance/beat.go:648    Beat ID: 64224213-06e2-4f42-8712-2d8af61efd1c
2020-10-06T10:40:36.792+1300    INFO    [beat]  instance/beat.go:976    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Elastic\\Beats\\7.9.2\\winlogbeat", "data": "C:\\Program Files\\Elastic\\Beats\\7.9.2\\winlogbeat\\data", "home": "C:\\Program Files\\Elastic\\Beats\\7.9.2\\winlogbeat", "logs": "C:\\Program Files\\Elastic\\Beats\\7.9.2\\winlogbeat\\logs"}, "type": "winlogbeat", "uuid": "64224213-06e2-4f42-8712-2d8af61efd1c"}}}
2020-10-06T10:40:36.792+1300    INFO    [beat]  instance/beat.go:985    Build info      {"system_info": {"build": {"commit": "2ab907f5ccecf9fd82fe37105082e89fd871f684", "libbeat": "7.9.2", "time": "2020-09-22T23:32:30.000Z", "version": "7.9.2"}}}
2020-10-06T10:40:36.798+1300    INFO    [beat]  instance/beat.go:988    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.14.7"}}}
2020-10-06T10:40:36.823+1300    INFO    [beat]  instance/beat.go:992    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-09-11T12:51:10.81+12:00","name":"NZ-CFG-11","ip":["fe80::bc70:7094:54a1:9bc1/64","10.150.16.37/24","fe80::1850:70ba:aab4:bb85/64","169.254.187.133/16","::1/128","127.0.0.1/8","fe80::5efe:a96:1025/128"],"kernel_version":"10.0.14393.3930 (rs1_release.200901-1914)","mac":["00:50:56:8e:39:79","00:ff:00:99:a5:37","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.3930"},"timezone":"NZDT","timezone_offset_sec":46800,"id":"cd66e37a-e62f-4a0a-a4fd-50786505f2a0"}}}
2020-10-06T10:40:36.828+1300    INFO    [beat]  instance/beat.go:1021   Process info    {"system_info": {"process": {"cwd": "C:\\Program Files\\Elastic\\Beats\\7.9.2\\winlogbeat", "exe": "C:\\Program Files\\Elastic\\Beats\\7.9.2\\winlogbeat\\winlogbeat.exe", "name": "winlogbeat.exe", "pid": 9460, "ppid": 6928, "start_time": "2020-10-06T10:40:36.570+1300"}}}
2020-10-06T10:40:36.839+1300    INFO    instance/beat.go:299    Setup Beat: winlogbeat; Version: 7.9.2
2020-10-06T10:40:36.914+1300    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'winlogbeat-7.9.2' as ILM is enabled.
2020-10-06T10:40:36.914+1300    INFO    eslegclient/connection.go:99    elasticsearch url: https://nz-sec-elastic.t3.nz:9200
2020-10-06T10:40:36.914+1300    INFO    [publisher]     pipeline/module.go:113  Beat name: NZ-CFG-11
2020-10-06T10:40:36.914+1300    INFO    beater/winlogbeat.go:69 State will be read from and persisted to C:\Program Files\Elastic\Beats\7.9.2\winlogbeat\data\.winlogbeat.yml
2020-10-06T10:40:36.982+1300    WARN    [cfgwarn]       registered_domain/registered_domain.go:60   BETA: The registered_domain processor is beta.
2020-10-06T10:40:37.063+1300    WARN    [cfgwarn]       registered_domain/registered_domain.go:60   BETA: The registered_domain processor is beta.
2020-10-06T10:40:37.077+1300    INFO    eslegclient/connection.go:99    elasticsearch url: https://nz-sec-elastic.t3.nz:9200
2020-10-06T10:40:37.107+1300    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.1
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2020-10-06T10:40:37.136+1300    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2020-10-06T10:40:37.136+1300    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-10-06T10:40:37.136+1300    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2020-10-06T10:40:37.136+1300    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{winlogbeat-7.9.2 {now/d}-000001}' as ILM is enabled.
2020-10-06T10:40:37.136+1300    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'winlogbeat-7.9.2-*' as ILM is enabled.
2020-10-06T10:40:37.136+1300    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {winlogbeat-7.9.2 {now/d}-000001} as ILM is enabled.
2020-10-06T10:40:37.136+1300    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {winlogbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-10-06T10:40:37.150+1300    INFO    template/load.go:169    Existing template will be overwritten, as overwrite is enabled.
2020-10-06T10:40:37.240+1300    INFO    template/load.go:109    Try loading template winlogbeat-7.9.2 to Elasticsearch
2020-10-06T10:40:37.291+1300    INFO    template/load.go:101    template with name 'winlogbeat-7.9.2' loaded.
2020-10-06T10:40:37.305+1300    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2020-10-06T10:40:37.311+1300    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
Index setup finished.

I am experiencing the same issue with Filebeat using the Crowdstrike or Fortinet module.

Further testing. I tried auditbeat but had an older version running which seems to work as expected

GET /auditbeat-7.7.1/_mapping/field/event.module

{
  "auditbeat-7.7.1-2020.10.06-000001" : {
    "mappings" : {
      "event.module" : {
        "full_name" : "event.module",
        "mapping" : {
          "module" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

Will upgrade auditbeat to latest and see if problem re-emerges.

@polyfractal @Andrew_G

Problem is resolved all round if I remove ...

setup.ilm.check_exists: false
1 Like

Thanks for sharing the solution @Derick_Jansen!

I'm following-up with the Beats team on it.

Thanks again for posting this @Derick_Jansen!

I was able to reproduce the invalid mappings with winlogbeat and auditbeat. Here's the issue that tracks it: https://github.com/elastic/beats/issues/21725

1 Like