Searching by Temporal Proximity

Elastic Stack Elasticsearch
1

I am quite new to ElasticSearch, most of what I need to seems to be easily
supported however I have hit one problem - specifically searching by
temporal proximity.

What do I mean by this? I'll explain a simplified scenario.

I have indexed events each with an associated date-time. I need to
discover those events that occur within a given time period of each other.
Ideally this time period would be arbitrary and specified at search time.

The "of each other" part is key here. I cannot just, say:

  • Aggregate by week or month, as an event that occurs at the end of one
    month is within a month of one which occurs at the start of the next month
    but would fall into a different bucket.
  • Do a date range based search, as I do not have a fixed range to search.

To describe what I need in a different way, in SQL I'd do something like:

SELECT eventid, eventdate from Events t1 where EXISTS(

SELECT 1 FROM Events t2 WHERE

t1.eventid <> t2.eventid AND

t1.eventdate BETWEEN

dateadd( day, -30, t2.eventdate )

AND

dateadd( day, 30, t2.eventdate )
);

The actual scenario is more complex - each event has a type and what I need
ultimately is to be able to answer questions of akin to:

"Find events of type X that occur with in 2 days of an event of type Y"

and even:

"Find events of type X that occur with in 2 days of an event of type Y and
of type Z"

Each event will be nested in/a child of, a parent record and I am only
interested in the temporal proximity of events with the same parent. The database
has a total of 10^9 events, each parent may have the order of 10^3
associated events. The use case is search heavy with ingests of deltas
approximately weekly.

I can munge the data on import in any way that would help. I have had a
couple of ideas on how to tackle the problem but neither are satisfactory.

I wondered whether there is a standard way to tackle this kind of
requirement in ElasticSearch and whether anyone else had run up against it.

Thanks,

Paul.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31bc7955-2043-49dd-865d-1dbc048a6dde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.