# Mail log event correlation **URL:** \ **Category:** Kibana\ **Created:** [September 2, 2021, 2:20am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115 "2021-09-02T02:20:23Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 2:20am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/1 "2021-09-02T02:20:23Z")
Apologies, I don't know whether this is the correct forum for this question, please bear with me. I have an elasticsearch index containing mail logging events from syslog, however as with postfix the complete transaction for an email is difficult to determine from an individual event and must be correlated together using the queue id across multiple logging events to piece the whole story together I'm used to doing this sort of log analysis manually with scripts I have developed, but I'm not sure even the terminology I could use to search for documentation on something like this in the elasticsearch ecosystem ... what should I be looking for? we're using td-agent/fluent to do log shipping from our fleet to our elasticsearch nodes and if this is something that needs to happen prior to elasticsearch indexing and then kibana performing searches then we can look at that - my assumption is that we might want to pull the relevant raw records from the index, do $magic, and then push that back into the index later to search on Any pointers appreciated, thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 2:40am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/2 "2021-09-02T02:40:42Z")
Welcome to our community! 😃 Do you want to roll all events for related IDs into a single thing? ---
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 2:42am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/3 "2021-09-02T02:42:51Z")
Mark - thanks for the reply, yes that would be ideal. There is a bit of extra work to tack on the initial connection as that is logged prior to the queue\_id being generated, but if we can collect events based on queue\_id that would be good enough for the moment. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 2:45am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/4 "2021-09-02T02:45:26Z")
Check out [Rolling up historical data | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/xpack-rollup.html) then, it might be what you want. ---
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 3:13am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/5 "2021-09-02T03:13:20Z")
Mark - looks interesting, will see how far I get there. Can I set those up from within the Kibana UI or can I only poke at the elasticsearch API? ---
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 3:17am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/6 "2021-09-02T03:17:32Z")
Mark - apologies, I found it and am working through that now, thanks for the assist ---
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 3:35am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/7 "2021-09-02T03:35:38Z")
Mark - I think this might not be what I'm after ... the intent is to collect events by their queue\_id but in a form that I can then search over rather than aggregate on, if I'm understanding Rollup properly - eg, in one line postfix logs the queue\_id and the sender of a message, in another it will be the queue\_id and the recipient(s) ... the only way to tie those together is to join them via the queue\_id value, but I'm looking to perform later searches based on sender or recipient ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 3:59am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/8 "2021-09-02T03:59:37Z")
No worries! In that case keeping each event is the best step, then just do a general search on the ID. That way each step is an individual log event that you can visualise. ---
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 4:58am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/9 "2021-09-02T04:58:29Z")
so really what you're suggesting is two searches - one to grab the queue\_id on a known field (such as recipient) and then search again to grab all events by those queue\_ids ... there's no way to create multi-event objects and search across those? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 5:02am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/10 "2021-09-02T05:02:59Z")
Ahh sorry, that's my fault. Take a look at [Tutorial: Transforming the eCommerce sample data | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/ecommerce-transforms.html), the concepts there of pivoting on customer ID are the same as mail ID. Is that better? ---
**Author:** ![mherbert](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@mherbert](https://discuss.elastic.co/u/mherbert)\ **Post date:** [September 2, 2021, 5:13am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/11 "2021-09-02T05:13:59Z")
nono, I don't know what words to use to describe the thing I'm after, but that does sound more like what I'm wanting ... will give that a shot, thanks for your patience ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 2, 2021, 5:25am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/12 "2021-09-02T05:25:51Z")
Yeah I get the challenge, I did stuff up with my original suggestions. KQL should be able to that with chaining, ie `queue_id: whatever | mail_id: whatever"`. But you still need an initial query to get the `queue_id`. I am not 100% sure there is something that can automatically tie those two together in one query. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 30, 2021, 5:26am UTC](https://discuss.elastic.co/t/mail-log-event-correlation/283115/13 "2021-09-30T05:26:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.