# Analytics system hits & visits **URL:** \ **Category:** Elasticsearch\ **Created:** [August 10, 2012, 4:13pm UTC](https://discuss.elastic.co/t/analytics-system-hits-visits/8707 "2012-08-10T16:13:42Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Alex\_P\_2](https://avatars.discourse-cdn.com/v4/letter/a/a9adbd/32.png) [@Alex\_P\_2](https://discuss.elastic.co/u/Alex_P_2)\ **Post date:** [August 10, 2012, 4:13pm UTC](https://discuss.elastic.co/t/analytics-system-hits-visits/8707/1 "2012-08-10T16:13:42Z")
Hi, I am building an analytics engine for documents. I am storing logs on each hit. I need a histogram of "hits" but also of "visits", the visits can be deduced by a session\_id. So multiple hits have the same session\_id. The mappings looks like: { "docHit" : { "properties" : { "doc\_id" : {"type" : "long", "index" : "not\_analyzed"}, "section\_id" : {"type" : "string", "index" : "not\_analyzed"}, ... } } } So I can return a histogram of "hits" for a particular document: { "query": { "term": { "doc\_id": 444 } }, "facets": { "hits\_per\_day": { "date\_histogram": { "field": "timestamp", "interval": "day" } } } } But how do I do the same for "visits"? If I wanted to get total visits for a document I could try: { "query": { "term": { "doc\_id": 444 } }, "facets": { "visits": { "terms": { "field": "session\_uid", size: 0 } } } } Which would return: { "took": 2, "timed\_out": false, "\_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 6, "max\_score": 0.0, "hits": [] }, "facets": { "visits": { "\_type": "terms", "missing": 0, "total": 6, "other": 0, "terms": [{ "term": "26A1473FFBF2CC5E3A8FCC9BF2240241", "count": 4 }, { "term": "3EC91387409740A1429676BB2A9CE02D", "count": 2 }] } } } But I would need to return ALL terms and compute the length of facets.visits.terms, which would be stupidly slow. Is there a straight forward way to tackle this use case? Thanks! Alex ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [August 11, 2012, 8:21pm UTC](https://discuss.elastic.co/t/analytics-system-hits-visits/8707/2 "2012-08-11T20:21:48Z")
Hi Alex, WARNING: this is sort of a "I'm having the same issue" reply. Not quite, but sort of. If I understand correctly, you want to know how many unique sessions you got, right? In that case you kinda need to count all the sessions, like you do in the stupidly slow facet you mentioned. Whether the facet is the best option I don't know, but you have to go through all of them somehow. I'm having a similar issue, and the only workaround I found is to do these facets incrementally. That is, because I have huge loads of data and I can't afford to do a facet on all of it. So what I do is: - keep a separate index/type where I intend to store my unique items - run a script regularly which will get a facet like the one you mentioned, but only for newly added data since the last run (I'm basing this on timestamp) - take the unique terms from there and insert them in the "unique items" type - because I have no idea whether any item is there already, I use another dirty hack: use the unique item itself as an ID and insert them with ?op\_type=create. So if it's already there it will give an error instead of inserting the same item twice - finally, when I want to do a statistic with those unique items I just run it on that "unique items" type. For example, if I want all my unique items since a certain date, it's just a matter of filtering by date and getting the number of hits. If anyone has a better option, I'd be delighted to hear it. On Friday, August 10, 2012 7:13:42 PM UTC+3, Alex P wrote: > Hi, > > I am building an analytics engine for documents. I am storing logs on each > hit. > > I need a histogram of "hits" but also of "visits", the visits can be > deduced by a session\_id. > > So multiple hits have the same session\_id. > > The mappings looks like: > { > "docHit" : { > "properties" : { > "doc\_id" : {"type" : "long", "index" : "not\_analyzed"}, > "section\_id" : {"type" : "string", "index" : "not\_analyzed"}, > ... > } > } > } > > So I can return a histogram of "hits" for a particular document: > { > "query": { > "term": { > "doc\_id": 444 > } > }, > "facets": { > "hits\_per\_day": { > "date\_histogram": { > "field": "timestamp", > "interval": "day" > } > } > } > } > > But how do I do the same for "visits"? > > If I wanted to get total visits for a document I could try: > { > "query": { > "term": { > "doc\_id": 444 > } > }, > "facets": { > "visits": { > "terms": { > "field": "session\_uid", > size: 0 > } > } > } > } > > Which would return: > { > "took": 2, > "timed\_out": false, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 6, > "max\_score": 0.0, > "hits": > }, > "facets": { > "visits": { > "\_type": "terms", > "missing": 0, > "total": 6, > "other": 0, > "terms": [{ > "term": "26A1473FFBF2CC5E3A8FCC9BF2240241", > "count": 4 > }, { > "term": "3EC91387409740A1429676BB2A9CE02D", > "count": 2 > }] > } > } > } > > But I would need to return ALL terms and compute the length of > facets.visits.terms, which would be stupidly slow. > > Is there a straight forward way to tackle this use case? > > Thanks! > > Alex -- ---
**Author:** ![Alex\_P\_2](https://avatars.discourse-cdn.com/v4/letter/a/a9adbd/32.png) [@Alex\_P\_2](https://discuss.elastic.co/u/Alex_P_2)\ **Post date:** [August 11, 2012, 8:27pm UTC](https://discuss.elastic.co/t/analytics-system-hits-visits/8707/3 "2012-08-11T20:27:52Z")
Thanks Radu! That's really interesting. I came up with my own little hack (which has some similarities to yours), I'm basically storing the data twice.... So I have a "hits" index, and then a "sessions" index. The sessions index has the exact same data, except it only has 1 entry per session. As opposed to having several entries per session (for the hits index) This means I'm duplicating data but now I can query for sessions just as easily as I can for hits. Which is necessary so I can find sessions per device, per country, per IP etc... And hopefully, if ES supports a better solutions, my "hits" index is intact and I can happily destroy all the "sessions" indexes. On Saturday, 11 August 2012 21:21:48 UTC+1, Radu Gheorghe wrote: > Hi Alex, > > WARNING: this is sort of a "I'm having the same issue" reply. Not quite, > but sort of. > > If I understand correctly, you want to know how many unique sessions you > got, right? > > In that case you kinda need to count all the sessions, like you do in the > stupidly slow facet you mentioned. Whether the facet is the best option I > don't know, but you have to go through all of them somehow. I'm having a > similar issue, and the only workaround I found is to do these facets > incrementally. That is, because I have huge loads of data and I can't > afford to do a facet on all of it. > > So what I do is: > > - keep a separate index/type where I intend to store my unique items > - run a script regularly which will get a facet like the one you > mentioned, but only for newly added data since the last run (I'm basing > this on timestamp) > - take the unique terms from there and insert them in the "unique items" > type > - because I have no idea whether any item is there already, I use > another dirty hack: use the unique item itself as an ID and insert them > with ?op\_type=create. So if it's already there it will give an error > instead of inserting the same item twice > > - finally, when I want to do a statistic with those unique items I just > run it on that "unique items" type. For example, if I want all my unique > items since a certain date, it's just a matter of filtering by date and > getting the number of hits. > > If anyone has a better option, I'd be delighted to hear it. > > On Friday, August 10, 2012 7:13:42 PM UTC+3, Alex P wrote: > > > Hi, > > > > I am building an analytics engine for documents. I am storing logs on > > each hit. > > > > I need a histogram of "hits" but also of "visits", the visits can be > > deduced by a session\_id. > > > > So multiple hits have the same session\_id. > > > > The mappings looks like: > > { > > "docHit" : { > > "properties" : { > > "doc\_id" : {"type" : "long", "index" : "not\_analyzed"}, > > "section\_id" : {"type" : "string", "index" : "not\_analyzed"}, > > ... > > } > > } > > } > > > > So I can return a histogram of "hits" for a particular document: > > { > > "query": { > > "term": { > > "doc\_id": 444 > > } > > }, > > "facets": { > > "hits\_per\_day": { > > "date\_histogram": { > > "field": "timestamp", > > "interval": "day" > > } > > } > > } > > } > > > > But how do I do the same for "visits"? > > > > If I wanted to get total visits for a document I could try: > > { > > "query": { > > "term": { > > "doc\_id": 444 > > } > > }, > > "facets": { > > "visits": { > > "terms": { > > "field": "session\_uid", > > size: 0 > > } > > } > > } > > } > > > > Which would return: > > { > > "took": 2, > > "timed\_out": false, > > "\_shards": { > > "total": 5, > > "successful": 5, > > "failed": 0 > > }, > > "hits": { > > "total": 6, > > "max\_score": 0.0, > > "hits": > > }, > > "facets": { > > "visits": { > > "\_type": "terms", > > "missing": 0, > > "total": 6, > > "other": 0, > > "terms": [{ > > "term": "26A1473FFBF2CC5E3A8FCC9BF2240241", > > "count": 4 > > }, { > > "term": "3EC91387409740A1429676BB2A9CE02D", > > "count": 2 > > }] > > } > > } > > } > > > > But I would need to return ALL terms and compute the length of > > facets.visits.terms, which would be stupidly slow. > > > > Is there a straight forward way to tackle this use case? > > > > Thanks! > > > > Alex -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [August 12, 2012, 9:05pm UTC](https://discuss.elastic.co/t/analytics-system-hits-visits/8707/4 "2012-08-12T21:05:43Z")
Hi Alex, I see Radu is already helping. 🙂 Plug: Should you want to use a search analytics service, have a look at > **[Cloud Monitoring Tools & Services | Sematext](https://sematext.com/cloud/)** > > Cloud monitoring solutions for smarter and faster DevOps ✓ Full-stack visibility into your infrastructure and application performance ✓ Start now! Otis On Friday, August 10, 2012 12:13:42 PM UTC-4, Alex P wrote: > Hi, > > I am building an analytics engine for documents. I am storing logs on each > hit. > > I need a histogram of "hits" but also of "visits", the visits can be > deduced by a session\_id. > > So multiple hits have the same session\_id. > > The mappings looks like: > { > "docHit" : { > "properties" : { > "doc\_id" : {"type" : "long", "index" : "not\_analyzed"}, > "section\_id" : {"type" : "string", "index" : "not\_analyzed"}, > ... > } > } > } > > So I can return a histogram of "hits" for a particular document: > { > "query": { > "term": { > "doc\_id": 444 > } > }, > "facets": { > "hits\_per\_day": { > "date\_histogram": { > "field": "timestamp", > "interval": "day" > } > } > } > } > > But how do I do the same for "visits"? > > If I wanted to get total visits for a document I could try: > { > "query": { > "term": { > "doc\_id": 444 > } > }, > "facets": { > "visits": { > "terms": { > "field": "session\_uid", > size: 0 > } > } > } > } > > Which would return: > { > "took": 2, > "timed\_out": false, > "\_shards": { > "total": 5, > "successful": 5, > "failed": 0 > }, > "hits": { > "total": 6, > "max\_score": 0.0, > "hits": > }, > "facets": { > "visits": { > "\_type": "terms", > "missing": 0, > "total": 6, > "other": 0, > "terms": [{ > "term": "26A1473FFBF2CC5E3A8FCC9BF2240241", > "count": 4 > }, { > "term": "3EC91387409740A1429676BB2A9CE02D", > "count": 2 > }] > } > } > } > > But I would need to return ALL terms and compute the length of > facets.visits.terms, which would be stupidly slow. > > Is there a straight forward way to tackle this use case? > > Thanks! > > Alex -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:16am UTC](https://discuss.elastic.co/t/analytics-system-hits-visits/8707/5 "2017-07-06T03:16:39Z")