Threat Intelligence View Not Using securitySolution:defaultThreatIndex

Elastic Security
1

I am working on developing threat intelligence integrations and have noticed that most integrations use the logs-ti_* naming format for both the original index (e.g., logs-ti_crowdstrike.ioc-default) and the transform destination index (e.g., logs-ti_crowdstrike_latest.dest_ioc-6). This causes problems when using the default logs-ti_* pattern because the pattern matches both indices - effectively duplicating effort on already resource intensive enrichments and visualizations.

I did some digging and found the securitySolution:defaultThreatIndex setting in Advanced Settings and changed this to the alias that points to the transform destination index. Now when I am creating threat intelligence rules, it uses the alias - great, but the IOC Intelligence view does not change. I am still seeing duplicate IOCs from both the original index and transform destination index in these results. This causes slow load times and inacurate metrics.

Additionally, the documentation says to not overwrite the default pattern:

By default, only the logs-ti_* index pattern is specified. Do not remove or overwrite this index pattern, as it is used by Elastic Agent integrations.

According to this, you should leave the logs-ti_* pattern, but then all Kibana features will be using duplicated data sets due to the issues described above, so this approach is confusing to me and does not seem to be efficient. It would make more sense to have Elastic’s threat intelligence integrations write to indices that do not match the logs-ti_* pattern and only reserve this pattern for transform destination indices or threat intelligence integrations that do not leverage transform jobs.

2

It looks like threat intelligence source indices are controlled with an ILM policy that has a hot tier max age of 1 day and delete min age of 2 days. So I guess while the data is duplicated for ~2 days, that is the best that can be achieved while using Elastic’s current approach.

Not too bad, but I still think there is room for improvement here. Specifically, it would be nice if the threat intelligence view respected the securitySolution:defaultThreatIndex setting or if the source indices did not use index names that match the logs-ti_* pattern.

For example: