# Need some ideas: Getting visits from hits out of logstash index **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2014, 9:12am UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493 "2014-07-06T09:12:28Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Stefan\_Hasenstab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefan_hasenstab/32/131812_2.png) [@Stefan\_Hasenstab](https://discuss.elastic.co/u/Stefan_Hasenstab)\ **Post date:** [July 6, 2014, 9:12am UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493/1 "2014-07-06T09:12:28Z")
Problem: I have aggregated accesslog data from different webservers in a large logstash index. My goal is to get the page _visits_ out of the accesslog hits. A _visit_ is defined as following: A visit results out of one or more hits from a single ip address in a specific time frame. Due to different products on the webservers each domain should be considered separately. My questions are: - Can this problem already be solved with build-in elasticsearch features? If _yes_, how? - If _no_: - What kind of plugin would you suggest? My own considerations lead from building a custom filter to retrieve just the data I need, to build a plugin which analyses the accesslog index and put the visit-data into a new index. Maybe someone can help me? I appreciate every answer. Thank you for your time! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2014, 9:27am UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493/2 "2014-07-06T09:27:48Z")
Are you using kibana? You should be able to extract this pretty simply if you are, if not, check it out. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 6 July 2014 19:12, Stefan Hasenstab [b00nb0b@gmail.com](mailto:b00nb0b@gmail.com) wrote: > Problem: > > I have aggregated accesslog data from different webservers in a large > logstash index. My goal is to get the page _visits_ out of the accesslog > hits. > > A _visit_ is defined as following: A visit results out of one or more > hits from a single ip address in a specific time frame. Due to different > products on the webservers each domain should be considered separately. > My questions are: > > - Can this problem already be solved with build-in elasticsearch > features? If _yes_, how? > - If _no_: > - What kind of plugin would you suggest? > > My own considerations lead from building a custom filter to retrieve just > the data I need, to build a plugin which analyses the accesslog index and > put the visit-data into a new index. > > Maybe someone can help me? I appreciate every answer. Thank you for your > time! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624beaqys7wyXm\_Ye5v37bPcZ9VROGV%2BSCLGh0MseWVsw9g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624beaqys7wyXm_Ye5v37bPcZ9VROGV%2BSCLGh0MseWVsw9g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Stefan\_Hasenstab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefan_hasenstab/32/131812_2.png) [@Stefan\_Hasenstab](https://discuss.elastic.co/u/Stefan_Hasenstab)\ **Post date:** [July 6, 2014, 12:08pm UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493/3 "2014-07-06T12:08:03Z")
Yes, I'm using kibana as well. Out of kibana i can manually extract this data, but the problem is that a SQL like "group by domain, ip" is not really doable on a large index. As far as I know anything with grouping involved is done internally with facets, which doesn't respect any kind of time filter. Am Sonntag, 6. Juli 2014 11:28:22 UTC+2 schrieb Mark Walkom: > Are you using kibana? You should be able to extract this pretty simply if > you are, if not, check it out. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 6 July 2014 19:12, Stefan Hasenstab \<[b00...@gmail.com](mailto:b00...@gmail.com) \\> > wrote: > > > Problem: > > > > I have aggregated accesslog data from different webservers in a large > > logstash index. My goal is to get the page _visits_ out of the accesslog > > hits. > > > > A _visit_ is defined as following: A visit results out of one or more > > hits from a single ip address in a specific time frame. Due to different > > products on the webservers each domain should be considered separately. > > My questions are: > > > > - Can this problem already be solved with build-in elasticsearch > > features? If _yes_, how? > > - If _no_: > > - What kind of plugin would you suggest? > > > > My own considerations lead from building a custom filter to retrieve just > > the data I need, to build a plugin which analyses the accesslog index and > > put the visit-data into a new index. > > > > Maybe someone can help me? I appreciate every answer. Thank you for your > > time! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/06924fcf-cd3e-4354-aa66-6e58428a9734%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/06924fcf-cd3e-4354-aa66-6e58428a9734%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Antonio\_Augusto\_Sant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonio_augusto_sant/32/82851_2.png) [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)\ **Post date:** [July 6, 2014, 1:15pm UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493/4 "2014-07-06T13:15:50Z")
DateHistogram aggregation can generate buckets by timeframe [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html) You probably want to aggregate by the page and latter aggregate by time or the oposite, what best suites your needs. On Sunday, July 6, 2014 9:08:03 AM UTC-3, Stefan wrote: > Yes, I'm using kibana as well. Out of kibana i can manually extract this > data, but the problem is that a SQL like "group by domain, ip" is not > really doable on a large index. As far as I know anything with grouping > involved is done internally with facets, which doesn't respect any kind of > time filter. > > Am Sonntag, 6. Juli 2014 11:28:22 UTC+2 schrieb Mark Walkom: > > > Are you using kibana? You should be able to extract this pretty simply if > > you are, if not, check it out. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 6 July 2014 19:12, Stefan Hasenstab [b00...@gmail.com](mailto:b00...@gmail.com) wrote: > > > > > Problem: > > > > > > I have aggregated accesslog data from different webservers in a large > > > logstash index. My goal is to get the page _visits_ out of the > > > accesslog hits. > > > > > > A _visit_ is defined as following: A visit results out of one or more > > > hits from a single ip address in a specific time frame. Due to different > > > products on the webservers each domain should be considered separately. > > > My questions are: > > > > > > - Can this problem already be solved with build-in elasticsearch > > > features? If _yes_, how? > > > - If _no_: > > > - What kind of plugin would you suggest? > > > > > > My own considerations lead from building a custom filter to retrieve > > > just the data I need, to build a plugin which analyses the accesslog index > > > and put the visit-data into a new index. > > > > > > Maybe someone can help me? I appreciate every answer. Thank you for your > > > time! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6beae3e9-1f11-4e36-983b-42bc1bdb5e42%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6beae3e9-1f11-4e36-983b-42bc1bdb5e42%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Stefan\_Hasenstab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefan_hasenstab/32/131812_2.png) [@Stefan\_Hasenstab](https://discuss.elastic.co/u/Stefan_Hasenstab)\ **Post date:** [July 6, 2014, 2:42pm UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493/5 "2014-07-06T14:42:55Z")
Ah nice, this looks exactly like what i need. But what is about memory consideration? The problem about histogram facets was, that all related data has to be loaded into memory, which is horrible if you want to group big data. ( [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-histogram-facet.html#_memory_considerations_3) ). Do you know how the new aggregation feature works internally? Am Sonntag, 6. Juli 2014 15:15:50 UTC+2 schrieb Antonio Augusto Santos: > DateHistogram aggregation can generate buckets by timeframe > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html) > > You probably want to aggregate by the page and latter aggregate by time or > the oposite, what best suites your needs. > > On Sunday, July 6, 2014 9:08:03 AM UTC-3, Stefan wrote: > > > Yes, I'm using kibana as well. Out of kibana i can manually extract this > > data, but the problem is that a SQL like "group by domain, ip" is not > > really doable on a large index. As far as I know anything with grouping > > involved is done internally with facets, which doesn't respect any kind of > > time filter. > > > > Am Sonntag, 6. Juli 2014 11:28:22 UTC+2 schrieb Mark Walkom: > > > > > Are you using kibana? You should be able to extract this pretty simply > > > if you are, if not, check it out. > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 6 July 2014 19:12, Stefan Hasenstab [b00...@gmail.com](mailto:b00...@gmail.com) wrote: > > > > > > > Problem: > > > > > > > > I have aggregated accesslog data from different webservers in a large > > > > logstash index. My goal is to get the page _visits_ out of the > > > > accesslog hits. > > > > > > > > A _visit_ is defined as following: A visit results out of one or more > > > > hits from a single ip address in a specific time frame. Due to different > > > > products on the webservers each domain should be considered separately. > > > > My questions are: > > > > > > > > - Can this problem already be solved with build-in elasticsearch > > > > features? If _yes_, how? > > > > - If _no_: > > > > - What kind of plugin would you suggest? > > > > > > > > My own considerations lead from building a custom filter to retrieve > > > > just the data I need, to build a plugin which analyses the accesslog index > > > > and put the visit-data into a new index. > > > > > > > > Maybe someone can help me? I appreciate every answer. Thank you for > > > > your time! > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1abed157-cdc2-4e0f-b314-a954c20b89f2%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dac3a78f-579b-42b9-b1b6-c93900a542b4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dac3a78f-579b-42b9-b1b6-c93900a542b4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:17am UTC](https://discuss.elastic.co/t/need-some-ideas-getting-visits-from-hits-out-of-logstash-index/18493/6 "2017-07-06T01:17:42Z")