# Alerting based on new docs based on saved users queries **URL:** \ **Category:** Elasticsearch\ **Created:** [November 4, 2010, 3:36pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517 "2010-11-04T15:36:26Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [November 4, 2010, 3:36pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/1 "2010-11-04T15:36:26Z")
Hello, We have a system where we check every new document that comes in against a large number of saved users queries. We do this, now, by running the \ AND . It isn't the most efficient setup, since for every doc, we need to run every saved user query. So, I there are two questions: 1. Within ES, is there a different approach that could be taken to streamline? 2. When running the USER\_QUERY AND NEWDOC query, what kind of caching makes the most sense? From my limited understanding, I think it'd make sense to filtercache both the USERQUERY and the NEWDOC query, since both of these should get repeated multiple times. Thanks! Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 4, 2010, 7:01pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/2 "2010-11-04T19:01:18Z")
Hey, On Thu, Nov 4, 2010 at 5:36 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Hello, > We have a system where we check every new document that comes in > against a large number of saved users queries. We do this, now, by > running the \ AND . It isn't the most > efficient setup, since for every doc, we need to run every saved user > query. What is exactly the NEWDOCUMENT part? How do you create it? > So, I there are two questions: > > 1. Within ES, is there a different approach that could be taken to > streamline? There are different ways to implement this internally, certainly with lower overhead than doing what you are doing now. One of the main problems here is the messaging aspect, which have different additional features. Can messages be lost? are duplicates allowed? If the client is down (registered to receive notifications), do docs need to be replayed for him? And, of course, actually implementing the notification aspect protocol wise.... . > 1. When running the USER\_QUERY AND NEWDOC query, what kind of caching > makes the most sense? From my limited understanding, I think it'd make > sense to filtercache both the USERQUERY and the NEWDOC query, since > both of these should get repeated multiple times. Depends on what the answer for NEWDOC is...., is it a term query with the doc id? > Thanks! > Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [November 4, 2010, 7:29pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/3 "2010-11-04T19:29:37Z")
Hey, Thanks for the followup and sorry for not being totally clear. Here is an explicit example: A user saves a search for "elasticsearch AND cool" to be alerted on. Every time a new document comes in, for every user query registered, we run against that doc, so a new doc id of 1 comes in, we end up with: (elasticsearch AND cool) AND (\_id:1) I'm considering wrapping each part in a filter query for the least overhead. At the moment these are two separate querystrings, but it'd be easy to update on my side to make the \_id match a termquery. Like I said, not efficient, but simple to understand and work with. If it matches we fire an alert. I believe that any of the alerting should be done from outside the ES realm and think (I really haven't looked, we have some custom setup) the alerting could be handled with a RabbitMQ setup or similar. What would be cool, but I have no idea the feasibility of would to: 1. Register queries against an index 2. When a document is indexed have details in the return citing which queries the document matched against 3. The application logic then handles any alerting or other actions Thanks, Paul On Nov 4, 1:01 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hey, > > On Thu, Nov 4, 2010 at 5:36 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > Hello, > > We have a system where we check every new document that comes in > > against a large number of saved users queries. We do this, now, by > > running the \ AND . It isn't the most > > efficient setup, since for every doc, we need to run every saved user > > query. > > What is exactly the NEWDOCUMENT part? How do you create it? > > > So, I there are two questions: > > > > 1. Within ES, is there a different approach that could be taken to > > streamline? > > There are different ways to implement this internally, certainly with lower > overhead than doing what you are doing now. One of the main problems here > is the messaging aspect, which have different additional features. Can > messages be lost? are duplicates allowed? If the client is down (registered > to receive notifications), do docs need to be replayed for him? And, of > course, actually implementing the notification aspect protocol wise.... . > > > 1. When running the USER\_QUERY AND NEWDOC query, what kind of caching > > makes the most sense? From my limited understanding, I think it'd make > > sense to filtercache both the USERQUERY and the NEWDOC query, since > > both of these should get repeated multiple times. > > Depends on what the answer for NEWDOC is...., is it a term query with the > doc id? > > > Thanks! > > Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 4, 2010, 7:44pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/4 "2010-11-04T19:44:26Z")
Hi, Regarding the filtering option, I suggest wrapping the alerting query in a filter, and use the \_id based matching as a term query. Make sure to prefix the id with the type name to give it type isolation (my\_type.\_id). No need to cache the \_id one as well. Returning on each indexed document if it matched or not is a good idea. The problem with how elasticsearch works now is that the relevant shard will have to be refreshed to in order for the indexed doc to be visible for search. Another option is to index it into a small in memory index, and run the queries on it, but this will incur indexing the doc twice (at least with the current way things are working). This feature certainly requires some more thinking :). -shay.banon On Thu, Nov 4, 2010 at 9:29 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Hey, > Thanks for the followup and sorry for not being totally clear. Here > is an explicit example: > A user saves a search for "elasticsearch AND cool" to be alerted on. > Every time a new document comes in, for every user query registered, > we run against that doc, so a new doc id of 1 comes in, we end up > with: > (elasticsearch AND cool) AND (\_id:1) > > I'm considering wrapping each part in a filter query for the least > overhead. At the moment these are two separate querystrings, but it'd > be easy to update on my side to make the \_id match a termquery. > > Like I said, not efficient, but simple to understand and work with. > > If it matches we fire an alert. > > I believe that any of the alerting should be done from outside the ES > realm and think (I really haven't looked, we have some custom setup) > the alerting could be handled with a RabbitMQ setup or similar. > What would be cool, but I have no idea the feasibility of would to: > > 1. Register queries against an index > 2. When a document is indexed have details in the return citing which > queries the document matched against > 3. The application logic then handles any alerting or other actions > > Thanks, > Paul > > On Nov 4, 1:01 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hey, > > > > On Thu, Nov 4, 2010 at 5:36 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > Hello, > > > We have a system where we check every new document that comes in > > > against a large number of saved users queries. We do this, now, by > > > running the \ AND . It isn't the most > > > efficient setup, since for every doc, we need to run every saved user > > > query. > > > > What is exactly the NEWDOCUMENT part? How do you create it? > > > > > So, I there are two questions: > > > > > > 1. Within ES, is there a different approach that could be taken to > > > streamline? > > > > There are different ways to implement this internally, certainly with > > lower > > overhead than doing what you are doing now. One of the main problems here > > is the messaging aspect, which have different additional features. Can > > messages be lost? are duplicates allowed? If the client is down > > (registered > > to receive notifications), do docs need to be replayed for him? And, of > > course, actually implementing the notification aspect protocol wise.... . > > > > > 1. When running the USER\_QUERY AND NEWDOC query, what kind of caching > > > makes the most sense? From my limited understanding, I think it'd make > > > sense to filtercache both the USERQUERY and the NEWDOC query, since > > > both of these should get repeated multiple times. > > > > Depends on what the answer for NEWDOC is...., is it a term query with the > > doc id? > > > > > Thanks! > > > Paul ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [November 4, 2010, 7:51pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/5 "2010-11-04T19:51:06Z")
Paul, Can you run these queries periodically every x seconds instead of for each new event? You'd need to track the last processed document and use that in the query. Something like "elasticsearch AND cool" and docId \> 5 (5 being the id of the last previously processed doc). There may be 1 or more matches, and you'd take action for each matching doc, etc. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Thu, Nov 4, 2010 at 3:29 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Hey, > Thanks for the followup and sorry for not being totally clear. Here > is an explicit example: > A user saves a search for "elasticsearch AND cool" to be alerted on. > Every time a new document comes in, for every user query registered, > we run against that doc, so a new doc id of 1 comes in, we end up > with: > (elasticsearch AND cool) AND (\_id:1) > > I'm considering wrapping each part in a filter query for the least > overhead. At the moment these are two separate querystrings, but it'd > be easy to update on my side to make the \_id match a termquery. > > Like I said, not efficient, but simple to understand and work with. > > If it matches we fire an alert. > > I believe that any of the alerting should be done from outside the ES > realm and think (I really haven't looked, we have some custom setup) > the alerting could be handled with a RabbitMQ setup or similar. > What would be cool, but I have no idea the feasibility of would to: > > 1. Register queries against an index > 2. When a document is indexed have details in the return citing which > queries the document matched against > 3. The application logic then handles any alerting or other actions > > Thanks, > Paul > > On Nov 4, 1:01 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hey, > > > > On Thu, Nov 4, 2010 at 5:36 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > Hello, > > > We have a system where we check every new document that comes in > > > against a large number of saved users queries. We do this, now, by > > > running the \ AND . It isn't the most > > > efficient setup, since for every doc, we need to run every saved user > > > query. > > > > What is exactly the NEWDOCUMENT part? How do you create it? > > > > > So, I there are two questions: > > > > > > 1. Within ES, is there a different approach that could be taken to > > > streamline? > > > > There are different ways to implement this internally, certainly with > > lower > > overhead than doing what you are doing now. One of the main problems here > > is the messaging aspect, which have different additional features. Can > > messages be lost? are duplicates allowed? If the client is down > > (registered > > to receive notifications), do docs need to be replayed for him? And, of > > course, actually implementing the notification aspect protocol wise.... . > > > > > 1. When running the USER\_QUERY AND NEWDOC query, what kind of caching > > > makes the most sense? From my limited understanding, I think it'd make > > > sense to filtercache both the USERQUERY and the NEWDOC query, since > > > both of these should get repeated multiple times. > > > > Depends on what the answer for NEWDOC is...., is it a term query with the > > doc id? > > > > > Thanks! > > > Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [November 5, 2010, 5:26pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/6 "2010-11-05T17:26:39Z")
Thanks for the feedback and ideas. There is definitely lots of room for improvement on how we are matching and there are trade offs with the number of registered users queries vs the doc volume, as well as, with the amount of state that needs to be stored. FWIW, another potential option in ES is to provide a callback to notify the app via thrift or HTTP, but I have pretty much no idea what I'm talking about here, in terms of feasibility or ES internals 🙂 Thanks, Paul On Nov 4, 1:51 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > Paul, > > Can you run these queries periodically every x seconds instead of for each > new event? You'd need to track the last processed document and use that in > the query. Something like "elasticsearch AND cool" and docId \> 5 (5 being > the id of the last previously processed doc). There may be 1 or more > matches, and you'd take action for each matching doc, etc. > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Thu, Nov 4, 2010 at 3:29 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > Hey, > > Thanks for the followup and sorry for not being totally clear. Here > > is an explicit example: > > A user saves a search for "elasticsearch AND cool" to be alerted on. > > Every time a new document comes in, for every user query registered, > > we run against that doc, so a new doc id of 1 comes in, we end up > > with: > > (elasticsearch AND cool) AND (\_id:1) > > > I'm considering wrapping each part in a filter query for the least > > overhead. At the moment these are two separate querystrings, but it'd > > be easy to update on my side to make the \_id match a termquery. > > > Like I said, not efficient, but simple to understand and work with. > > > If it matches we fire an alert. > > > I believe that any of the alerting should be done from outside the ES > > realm and think (I really haven't looked, we have some custom setup) > > the alerting could be handled with a RabbitMQ setup or similar. > > What would be cool, but I have no idea the feasibility of would to: > > > > 1. Register queries against an index > > 2. When a document is indexed have details in the return citing which > > queries the document matched against > > 3. The application logic then handles any alerting or other actions > > > Thanks, > > Paul > > > On Nov 4, 1:01 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hey, > > > > On Thu, Nov 4, 2010 at 5:36 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > > > Hello, > > > > We have a system where we check every new document that comes in > > > > against a large number of saved users queries. We do this, now, by > > > > running the \ AND . It isn't the most > > > > efficient setup, since for every doc, we need to run every saved user > > > > query. > > > > What is exactly the NEWDOCUMENT part? How do you create it? > > > > > So, I there are two questions: > > > > > > > > 1. Within ES, is there a different approach that could be taken to > > > > streamline? > > > > There are different ways to implement this internally, certainly with > > > lower > > > overhead than doing what you are doing now. One of the main problems here > > > is the messaging aspect, which have different additional features. Can > > > messages be lost? are duplicates allowed? If the client is down > > > (registered > > > to receive notifications), do docs need to be replayed for him? And, of > > > course, actually implementing the notification aspect protocol wise.... . > > > > > 1. When running the USER\_QUERY AND NEWDOC query, what kind of caching > > > > makes the most sense? From my limited understanding, I think it'd make > > > > sense to filtercache both the USERQUERY and the NEWDOC query, since > > > > both of these should get repeated multiple times. > > > > Depends on what the answer for NEWDOC is...., is it a term query with the > > > doc id? > > > > > Thanks! > > > > Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 5, 2010, 7:35pm UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/7 "2010-11-05T19:35:54Z")
On Fri, Nov 5, 2010 at 7:26 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Thanks for the feedback and ideas. There is definitely lots of room > for improvement on how we are matching and there are trade offs with > the number of registered users queries vs the doc volume, as well as, > with the amount of state that needs to be stored. > > FWIW, another potential option in ES is to provide a callback to > notify the app via thrift or HTTP, but I have pretty much no idea what > I'm talking about here, in terms of feasibility or ES internals 🙂 It is possible, everything is possible :). The point here is once ES will provide callbacks, it gets into the messaging aspect, where what happens if a client disconnects, does ES provides reliable notifications replaying missing notifications, and so on. This is very different from returning the matched queries on index operations (simpler to implement, more burden on the client). > Thanks, > Paul > > On Nov 4, 1:51 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > > > Paul, > > > > Can you run these queries periodically every x seconds instead of for > > each > > new event? You'd need to track the last processed document and use that > > in > > the query. Something like "elasticsearch AND cool" and docId \> 5 (5 being > > the id of the last previously processed doc). There may be 1 or more > > matches, and you'd take action for each matching doc, etc. > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Thu, Nov 4, 2010 at 3:29 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > Hey, > > > Thanks for the followup and sorry for not being totally clear. Here > > > is an explicit example: > > > A user saves a search for "elasticsearch AND cool" to be alerted on. > > > Every time a new document comes in, for every user query registered, > > > we run against that doc, so a new doc id of 1 comes in, we end up > > > with: > > > (elasticsearch AND cool) AND (\_id:1) > > > > > I'm considering wrapping each part in a filter query for the least > > > overhead. At the moment these are two separate querystrings, but it'd > > > be easy to update on my side to make the \_id match a termquery. > > > > > Like I said, not efficient, but simple to understand and work with. > > > > > If it matches we fire an alert. > > > > > I believe that any of the alerting should be done from outside the ES > > > realm and think (I really haven't looked, we have some custom setup) > > > the alerting could be handled with a RabbitMQ setup or similar. > > > What would be cool, but I have no idea the feasibility of would to: > > > > > > 1. Register queries against an index > > > 2. When a document is indexed have details in the return citing which > > > queries the document matched against > > > 3. The application logic then handles any alerting or other actions > > > > > Thanks, > > > Paul > > > > > On Nov 4, 1:01 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Hey, > > > > > > On Thu, Nov 4, 2010 at 5:36 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > > > > > Hello, > > > > > We have a system where we check every new document that comes in > > > > > against a large number of saved users queries. We do this, now, by > > > > > running the \ AND . It isn't the most > > > > > efficient setup, since for every doc, we need to run every saved > > > > > user > > > > > query. > > > > > > What is exactly the NEWDOCUMENT part? How do you create it? > > > > > > > So, I there are two questions: > > > > > > > > > > 1. Within ES, is there a different approach that could be taken to > > > > > streamline? > > > > > > There are different ways to implement this internally, certainly with > > > > lower > > > > overhead than doing what you are doing now. One of the main problems > > > > here > > > > is the messaging aspect, which have different additional features. > > > > Can > > > > messages be lost? are duplicates allowed? If the client is down > > > > (registered > > > > to receive notifications), do docs need to be replayed for him? And, > > > > of > > > > course, actually implementing the notification aspect protocol > > > > wise.... . > > > > > > > 1. When running the USER\_QUERY AND NEWDOC query, what kind of > > > > > caching > > > > > makes the most sense? From my limited understanding, I think it'd > > > > > make > > > > > sense to filtercache both the USERQUERY and the NEWDOC query, since > > > > > both of these should get repeated multiple times. > > > > > > Depends on what the answer for NEWDOC is...., is it a term query with > > > > the > > > > doc id? > > > > > > > Thanks! > > > > > Paul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:17am UTC](https://discuss.elastic.co/t/alerting-based-on-new-docs-based-on-saved-users-queries/3517/8 "2017-07-06T04:17:02Z")