I am quite new to ElasticSearch, most of what I need to seems to be easily
supported however I have hit one problem - specifically searching by
temporal proximity.
What do I mean by this? I'll explain a simplified scenario.
I have indexed events each with an associated date-time. I need to
discover those events that occur within a given time period of each other.
Ideally this time period would be arbitrary and specified at search time.
The "of each other" part is key here. I cannot just, say:
- Aggregate by week or month, as an event that occurs at the end of one
month is within a month of one which occurs at the start of the next month
but would fall into a different bucket. - Do a date range based search, as I do not have a fixed range to search.
To describe what I need in a different way, in SQL I'd do something like:
SELECT eventid, eventdate from Events t1 where EXISTS(
SELECT 1 FROM Events t2 WHERE
t1.eventid <> t2.eventid AND
t1.eventdate BETWEEN
dateadd( day, -30, t2.eventdate )
AND
dateadd( day, 30, t2.eventdate )
);
The actual scenario is more complex - each event has a type and what I need
ultimately is to be able to answer questions of akin to:
"Find events of type X that occur with in 2 days of an event of type Y"
and even:
"Find events of type X that occur with in 2 days of an event of type Y and
of type Z"
Each event will be nested in/a child of, a parent record and I am only
interested in the temporal proximity of events with the same parent. The database
has a total of 10^9 events, each parent may have the order of 10^3
associated events. The use case is search heavy with ingests of deltas
approximately weekly.
I can munge the data on import in any way that would help. I have had a
couple of ideas on how to tackle the problem but neither are satisfactory.
I wondered whether there is a standard way to tackle this kind of
requirement in ElasticSearch and whether anyone else had run up against it.
Thanks,
Paul.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31bc7955-2043-49dd-865d-1dbc048a6dde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.