Lots of unmapped fields in .siem-signals-default

[willemdh]

Hello,

There seem to be quite a bit of unmapped fields in .siem-signals-default index. I tried refreshing the index pattern, but the fields are not added. The result is I can't use certain fields:

For example a search for

transaction.type: request

Doesn't give me any results. These fields also do not appear in the index pattern:

Grtz

Willem

[Frank_Hassanabad]

Hi @willemdh,

In detection engine we do not add the ECS extended fields as part of the mapping such as parts of the transaction ones:
https://www.elastic.co/guide/en/ecs/current/ecs-tracing.html

However, if you want them indexed you can add them either directly or better yet to the template so that any newly created indexes from the ILM policy will take effect.

You can change the template/mapping to whichever space you want to and then any newly created data will be indexed from that point forward from that space. You can create different template indexing and life cycle management policies per each space you want to as well since siem signals are per space.

To see the ILM policy, the current mapping, and the current template for the default space from dev tools would be this:

GET .siem-signals-default/_ilm/explain
GET .siem-signals-default/_mapping
GET _template/.siem-signals-default
GET _ilm/policy/.siem-signals-default

If you are using another space such as "test-space", then replace ".siem-signals-default" with ".siem-signals-test-space" to see the settings for that space above.

To change any of the settings such as the template for newly created indexes when a roll over occurs and then the existing mapping you can do from dev tools using these references:

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

I don't know how sensitive your current data set is, or your production environment so I have to suffix all of this with please backup your siem signals, templates, mappings, etc... before doing any of these operations in case you need to "roll back" any changes and feel comfortable with the tooling:
https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

On upgrades our product if you go this route, you just have to be cautious as we might migrate the indexes for you forward but we will keep in mind that advanced users might have "hand edited" some of these settings. So on future upgrades you will just want to double check that your custom mappings have rolled forward with our product or not and perform regular backups of your data using snapshots.

[willemdh]

@Frank_Hassanabad Tx for the detailed explanation. I'm considering going down the following road (which I'm doing for beat indices too for multiple years and survives updates)

Create an 'overrides' template and add custom and extended fields there which points to the ".siem-signals-default-*" index and with a higher order then the builtin template

{
  ".siem-signals-default-overrides" : {
    "order" : 99,
    "index_patterns" : [
      ".siem-signals-default-*"
    ], 

This might prevent issues when updating Elastic?

Do you think this is a good approach?

Grtz

Willem

[Frank_Hassanabad]

That's not a bad idea. In the code we always look for .siem-signals-${space-id} such as .siem-signals-default within SIEM if that helps you out planning a strategy.

Happy to hear you're a very experienced beats person!