Timeline Template not applied when Alert fires

[PhilA (Phil)]

Hi

I am having issues with timeline templates. I have built a custom template showing some specific fields I would be interested in for blocked sessions thorugh a firewall.

The timeline is configured and in the timeline view it looks good. I use a template field to filter on the source IP.

That timeline template is saved and I have edited the Alert rule to now use the template, however when I view a new alert and click the timeline it still uses the default template and columns.

I'm running the latest version, v8.1.2.

Has anyone else had this issue or good suggestions of how I can investigate?

Thanks

[Mike_Paquette]

Hi @PhilA, Great to see you're trying out Timeline Templates. I use custom Timeline templates all the time, and I have them working on my 8.1 system.

Please excuse the simplicity of this question, but did you try disabling and re-activating your rule after editing it to add your new custom timeline template?

-Mike P.

[PhilA (Phil)]

Hi @Mike_Paquette

Thanks for the message and I will take any suggestions, no matter how simple!

Yes I did disable and re-enable the rule and have also re-created it completely.

One thing I did just test was changing the filter. I added an element to the filter query where the red circle is, just adding a broad range source IP so it would always match but this verified that the Timeline Template IS being used, its just that the columns are not being applied. Below shows my template so note the columns listed includes source and destination bytes.

When the alert fires, you can see the filter is there but the columns don't match, missing the bytes

It seems it is using part but not all of the properties of my template.

Phil

[spong (Garrett)]

Thanks for the details @PhilA! I just tried to reproduce locally and wasn't able to. Would you be willing to export your Timeline Template and share it with us to test? Perhaps there's something specific within your template that is resulting in this behavior?

You can export from the Timelines page and then click the Templates tab, select your Timeline Template, then Bulk actions -> Export selected:

Thanks!
Garrett

[PhilA (Phil)]

Thanks for your support Garrett (@spong). I can't seem to attach the export as an ndjson file as it seems to only allow for image uploads so the json code is below, hopefully you can just copy that in to an ndjson file and import that? Trying to view the JSON myself I think it looks like the columns are there but this isn't my skillset!

If its easier for me to share this with you another way let me know.

Appreciate your help

Phil

{"savedObjectId":"c548bf70-b654-11ec-8a31-7ba9a7f2f52c","version":"WzE5MjE4NSwyXQ==","columns":[{"columnHeaderType":"not-filtered","id":"@timestamp","type":"number"},{"columnHeaderType":"not-filtered","id":"event.action"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"source.user.name","category":"source","type":"string","example":"a.einstein"},{"columnHeaderType":"not-filtered","id":"source.ip"},{"aggregatable":true,"description":"Bytes sent from the source to the destination.","columnHeaderType":"not-filtered","id":"source.bytes","category":"source","type":"number"},{"columnHeaderType":"not-filtered","id":"destination.ip"},{"aggregatable":true,"description":"Port of the destination.","columnHeaderType":"not-filtered","id":"destination.port","category":"destination","type":"number"},{"aggregatable":true,"description":"Bytes sent from the destination to the source.","columnHeaderType":"not-filtered","id":"destination.bytes","category":"destination","type":"number"}],"dataProviders":[{"excluded":false,"and":[],"kqlQuery":"","name":"{source.ip}","queryMatch":{"field":"source.ip","value":"{source.ip}","operator":":"},"id":"timeline-1-1210648b-8456-4012-8fae-92597b54c9c7","type":"template","enabled":true}],"description":"View outbound blocked traffic through the Palo","eventType":"all","filters":[],"kqlMode":"filter","timelineType":"template","kqlQuery":{"filterQuery":{"serializedQuery":"{\"bool\":{\"filter\":[{\"bool\":{\"should\":[{\"match\":{\"event.module\":\"panw\"}}],\"minimum_should_match\":1}},{\"bool\":{\"should\":[{\"bool\":{\"should\":[{\"match\":{\"event.action\":\"flow_dropped\"}}],\"minimum_should_match\":1}},{\"bool\":{\"should\":[{\"match\":{\"event.action\":\"flow_denied\"}}],\"minimum_should_match\":1}}],\"minimum_should_match\":1}}]}}","kuery":{"expression":"event.module : panw and event.action : (flow_dropped or flow_denied) ","kind":"kuery"}}},"title":"Palo Outbound Blocked","sort":[{"columnType":"number","sortDirection":"desc","columnId":"@timestamp"}],"templateTimelineId":"2829c772-fe8e-470c-bac4-7b960ac797a1","templateTimelineVersion":1,"created":1649324049393,"createdBy":"soc","updated":1649424529242,"updatedBy":"soc","dateRange":{"start":"2022-04-06T09:10:18.488Z","end":"2022-04-07T09:10:18.488Z"},"indexNames":["filebeat-*"],"eqlOptions":{"tiebreakerField":"","size":100,"query":"","eventCategoryField":"event.category","timestampField":"@timestamp"},"favorite":[],"savedQueryId":null,"dataViewId":"filebeat-*","eventNotes":[],"globalNotes":[],"pinnedEventIds":[]}

[spong (Garrett)]

Thanks for supplying the template @PhilA, this is really helpful!

I was able to get it loaded, and it seems to be loading the columns and query for me, but is actually erroring out when trying to load the alert -- both a query error toast is displayed (presumably from trying to append the _id of the alert), and it also doesn't fill in the source.ip field either:

I'm not super familiar with the timeline/template side of things, but let me discuss with someone from that team and I'll get back to you. Definitely looks like a bug on our end though, but hopefully we can find a workaround for you while we work a fix.

Thanks again for all the details here!

[PhilA (Phil)]

Thanks Garrett

I think that error is because the template field isn't working so its trying to include the source.ip value in the filter but its broken so when put together, the entire filter query is broken.

That aside, I think you have the same state as me - it looks correct in the Timelines view but when an alert fires that utilises that Timeline template, the timeline doesn't apply correctly and misses out the columns.

I must stress I am heading out of my skill area here but potentially some useful information...

If I click the timeline button on an alert which uses this timeline template with the Google Chrome Dev window running I see the following :

• I can see a request in the HTTP headers for the timeline template reporting a template ID of 2829c772-fe8e-470c-bac4-7b960ac797a1. This matches the ID of the timeline I have created so this looks good. Request URL is https://soc-elastic.comm.ad.roke.co.uk:5601/api/timeline?template_timeline_id=2829c772-fe8e-470c-bac4-7b960ac797a1

• The response also looks sensible, shows HTTP 200 OK and the response includes the correct fields (I have cut some off this response as it was long but you can see the Bytes sent column being referenced at the end of this code); {"data":{"getOneTimeline":{"savedObjectId":"c548bf70-b654-11ec-8a31-7ba9a7f2f52c","version":"WzE5MjE4NSwyXQ==","columns":[{"columnHeaderType":"not-filtered","id":"@timestamp","type":"number"},{"columnHeaderType":"not-filtered","id":"event.action"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"source.user.name","category":"source","type":"string","example":"a.einstein"},{"columnHeaderType":"not-filtered","id":"source.ip"},{"aggregatable":true,"description":"Bytes sent from the source to the destination."...

The preview field also confirms the fields in the response so up to this stage, all seems to work perfectly.

Looking at the screenshot below, all of the above data taken from the data at the request circled in red. Going down to the request circled in blue and the fields are back to the wrong ones. Presumably this is the request that is actually presented to me.

I don't know if this helps but hopefully it does. If I can gather any more data or logs please let me know

Thanks

Phil

[spong (Garrett)]

Hey there @PhilA, thanks for all the additional details!

I had a big long message typed up to you with additional debug steps as I was unable to initially reproduce and then I found exactly what I was looking for when I did one more lap through the code...

This isn't happening on a Threshold Rule by chance, is it?? If so, we've found the culprit and I've opened this issue for tracking. Appears Threshold Rule Alerts's have special logic when duplicating from the template fields and columns aren't being included there. I was able to confirm this behavior retesting on 8.1.2 using a Threshold Rule type with your provided template:

---

If this happening for you and is not a Threshold Rule we can try to debug further, but this looks to be the issue at play here. I'm not seeing a workaround unfortunately, but this should be a pretty straightforward fix so I requested on the issue that it be backported as far back as it can so hopefully a minor upgrade won't be required. You can follow along with that ticket for what version the fix lands in though.

All that said, below is some of my initial message that's still relevant, and some other issues I created as part of this investigation. Thanks again for reporting this @PhilA, and for all the details provided as well!

Cheers!
Garrett

---

So I had a chance to chat with some of the Timeline/Template folks today and have a bit more information.

So first off, with regards to the extra requests you were seeing when using the Investigate in Timeline action, those are all okay/expected, and correspond to the creation of a new timeline from the Template provided when using the Investigate action. Basically it's fetching the template, then creating a new timeline from said template, so since you're seeing the new timeline not have the new fields we can suspect the bug is somewhere between the template being read and creating this new timeline. Here's the requests annotated:

(it was about here I found the exact spot of the issue )

And in discussions/testing with the Timeline folks earlier today, I've created these two issues and an enhancement to improve some of the UX around templates:

These two bugs:

• [Security Solution][Timeline Templates] Fallback to default Timeline if Timeline Template has been deleted
• [Security Solution][Timeline Templates] KQLSyntaxError when `Template Field` is not present on Alert document · Issue #129958 · elastic/kibana · GitHub

And this enhancement:

• [Security Solution][Timeline Templates] Always add alert _id Data Provider and select Alerts Data View for all Templates by default

As a heads up, (with regards to the above enhancement), you may want to update your Data View to include the Alerts Index, otherwise your Timeline won't be able to include any alert data. Adding a template Data Provider for _id: {_id} (as an OR) will also ensure that the alert document shows up in the table.

Cheers!
Garrett

[PhilA (Phil)]

Hi Garrett

Thank you, it is a Threshold alert so that would confirm the issue

Thanks for all the detail, I'll update my current timeline and keep an eye on the issue to make sure I get upgrades once it is available.

I appreciate your support

Phil