# \[ANN\] SegmentSpy - Site Plugin to watch segments in realtime **URL:** \ **Category:** Elasticsearch\ **Created:** [February 3, 2013, 2:54pm UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608 "2013-02-03T14:54:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [February 3, 2013, 2:54pm UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/1 "2013-02-03T14:54:14Z")
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges[http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)", written by Mike McCandless on his blog. Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies. ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy[https://github.com/polyfractal/elasticsearch-segmentspy](https://github.com/polyfractal/elasticsearch-segmentspy) The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count. ES provides the physical size of a segment but it doesn't specify how big the deleted\_docs section is (just number of deleted docs). ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did. Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data. =) -Zach -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Michael\_McCandless](https://avatars.discourse-cdn.com/v4/letter/m/e47c2d/32.png) [@Michael\_McCandless](https://discuss.elastic.co/u/Michael_McCandless)\ **Post date:** [February 4, 2013, 12:36am UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/2 "2013-02-04T00:36:46Z")
Very cool! For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs. Mike [http://blog.mikemccandless.com](http://blog.mikemccandless.com) On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) wrote: > A while ago, Lukas showed me an article titled "Visualizing Lucene's > Segment Merges[http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)", > written by Mike McCandless on his blog. Since reading that article I've > been enamored with the segment visualizations that he created to > demonstrate different merge policies. > > Elasticsearch provides some of the necessary information through the Index > Segments API, so I created a fairly simple site plugin to graph this data: > Elasticsearch-SegmentSpy[https://github.com/polyfractal/elasticsearch-segmentspy](https://github.com/polyfractal/elasticsearch-segmentspy) > > The plugin is still very rough around the edges, and has only been tested > on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In > Mike's article, he graphed segments by physical size, while my plugin > graphs by doc count. ES provides the physical size of a segment but it > doesn't specify how big the deleted\_docs section is (just number of deleted > docs). ES also doesn't expose which segments are merging, so I can't color > actively merging segments like Mike did. > > Honestly, I doubt this plugin is very practical or useful...but it is fun > to watch segments merge as you index/delete from your data. =) > -Zach > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [February 4, 2013, 12:51pm UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/3 "2013-02-04T12:51:10Z")
Ahh, that makes sense. I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =) Thanks for taking a look at the plugin! -Zach On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote: > Very cool! > > For the deleted docs in the videos I think I just pro-rated according to > %tg deleted docs. > > Mike > > [http://blog.mikemccandless.com](http://blog.mikemccandless.com) > > On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong \<[zachar...@gmail.com](mailto:zachar...@gmail.com)\ > > > wrote: > > > A while ago, Lukas showed me an article titled "Visualizing Lucene's > > Segment Merges[http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)", > > written by Mike McCandless on his blog. Since reading that article I've > > been enamored with the segment visualizations that he created to > > demonstrate different merge policies. > > > > Elasticsearch provides some of the necessary information through the > > Index Segments API, so I created a fairly simple site plugin to graph this > > data: Elasticsearch-SegmentSpy[https://github.com/polyfractal/elasticsearch-segmentspy](https://github.com/polyfractal/elasticsearch-segmentspy) > > > > The plugin is still very rough around the edges, and has only been tested > > on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In > > Mike's article, he graphed segments by physical size, while my plugin > > graphs by doc count. ES provides the physical size of a segment but it > > doesn't specify how big the deleted\_docs section is (just number of deleted > > docs). ES also doesn't expose which segments are merging, so I can't color > > actively merging segments like Mike did. > > > > Honestly, I doubt this plugin is very practical or useful...but it is fun > > to watch segments merge as you index/delete from your data. =) > > -Zach > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 12, 2013, 10:29pm UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/4 "2013-02-12T22:29:40Z")
Very nice!. We will check if we can add a flag to the segments API to indicate that its begin merged. Can you open an issue? On Feb 4, 2013, at 1:51 PM, Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) wrote: > Ahh, that makes sense. I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =) > > Thanks for taking a look at the plugin! > -Zach > > On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote: > Very cool! > > For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs. > > Mike > > [http://blog.mikemccandless.com](http://blog.mikemccandless.com) > > On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong [zachar...@gmail.com](mailto:zachar...@gmail.com) wrote: > A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog. Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies. > > Elasticsearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: Elasticsearch-SegmentSpy > > The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count. ES provides the physical size of a segment but it doesn't specify how big the deleted\_docs section is (just number of deleted docs). ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did. > > Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data. =) > -Zach > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [February 12, 2013, 10:47pm UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/5 "2013-02-12T22:47:46Z")
I've really enjoyed this plugin, it makes watching a large Bulk load interesting. Even with our simple 3-node, 5 shard cluster though one can't see the 'whole picture'. This is where a 3-D grid column chart would look very cool. Each Index could be a grid, with extra z-axis used to display the other shards. So X-axis is each segment as it is now, and the Y-axis is the logarithmic size as it is now, but then plot each other shard via the z-axis. This way one chart for each index can be seen on one page. Obviously your JS chart library may not have that chart feature though, but I think it would look awesome, and be more practical for even relatively large indices? I tried mocking this up in Excel but sadly it thinks it's too complex to display even just 2 series with 5 rows.. Stupid Excel... ? something looking like this anyway: [http://www.agocg.ac.uk/reports/graphics/34/appii97/image18.gif](http://www.agocg.ac.uk/reports/graphics/34/appii97/image18.gif) as an example. cheers, Paul On 13 February 2013 09:29, [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Very nice!. We will check if we can add a flag to the segments API to > indicate that its begin merged. Can you open an issue? > > On Feb 4, 2013, at 1:51 PM, Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com) wrote: > > Ahh, that makes sense. I'll make some changes when I get a chance to > account for that...I'd be happier if the segments represented size instead > of docs. =) > > Thanks for taking a look at the plugin! > -Zach > > On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote: > > > Very cool! > > > > For the deleted docs in the videos I think I just pro-rated according to > > %tg deleted docs. > > > > Mike > > > > [http://blog.mikemccandless.com](http://blog.mikemccandless.com) > > > > On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong [zachar...@gmail.com](mailto:zachar...@gmail.com) wrote: > > > > > A while ago, Lukas showed me an article titled "Visualizing Lucene's > > > Segment Merges[http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)", > > > written by Mike McCandless on his blog. Since reading that article I've > > > been enamored with the segment visualizations that he created to > > > demonstrate different merge policies. > > > > > > Elasticsearch provides some of the necessary information through the > > > Index Segments API, so I created a fairly simple site plugin to graph this > > > data: Elasticsearch-SegmentSpy[https://github.com/polyfractal/elasticsearch-segmentspy](https://github.com/polyfractal/elasticsearch-segmentspy) > > > > > > The plugin is still very rough around the edges, and has only been > > > tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any > > > errors. In Mike's article, he graphed segments by physical size, while my > > > plugin graphs by doc count. ES provides the physical size of a segment but > > > it doesn't specify how big the deleted\_docs section is (just number of > > > deleted docs). ES also doesn't expose which segments are merging, so I > > > can't color actively merging segments like Mike did. > > > > > > Honestly, I doubt this plugin is very practical or useful...but it is > > > fun to watch segments merge as you index/delete from your data. =) > > > -Zach > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [February 12, 2013, 11:27pm UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/6 "2013-02-12T23:27:06Z")
> Very nice!. We will check if we can add a flag to the segments API to > indicate that its begin merged. Can you open an issue? Thanks! I'll go open an issue right now...that'd be a cool feature to include in the plugin. Even with our simple 3-node, 5 shard cluster though one can't see the > 'whole picture'. This is where a 3-D grid column chart would look very cool. Yeah, I totally agree. My cluster has 3 nodes as well, and it quickly becomes unmanageable to watch all the indices. I experimented with making all the graphs smaller (ala BigDesk), and it sorta works, but is still difficult to get everything on screen at a reasonable resolution. I'll play around with 3D charts, that's an excellent idea. Karmi has convinced me to switch to D3, so more advanced visualizations should definitely be possible. Thanks for the comments! -Zach On Tuesday, February 12, 2013 5:29:40 PM UTC-5, kimchy wrote: > Very nice!. We will check if we can add a flag to the segments API to > indicate that its begin merged. Can you open an issue? > > On Feb 4, 2013, at 1:51 PM, Zachary Tong \<[zachar...@gmail.com](mailto:zachar...@gmail.com)\\> > wrote: > > Ahh, that makes sense. I'll make some changes when I get a chance to > account for that...I'd be happier if the segments represented size instead > of docs. =) > > Thanks for taking a look at the plugin! > -Zach > > On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote: > > > Very cool! > > > > For the deleted docs in the videos I think I just pro-rated according to > > %tg deleted docs. > > > > Mike > > > > [http://blog.mikemccandless.com](http://blog.mikemccandless.com) > > > > On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong [zachar...@gmail.com](mailto:zachar...@gmail.com) wrote: > > > > > A while ago, Lukas showed me an article titled "Visualizing Lucene's > > > Segment Merges[http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)", > > > written by Mike McCandless on his blog. Since reading that article I've > > > been enamored with the segment visualizations that he created to > > > demonstrate different merge policies. > > > > > > Elasticsearch provides some of the necessary information through the > > > Index Segments API, so I created a fairly simple site plugin to graph this > > > data: Elasticsearch-SegmentSpy[https://github.com/polyfractal/elasticsearch-segmentspy](https://github.com/polyfractal/elasticsearch-segmentspy) > > > > > > The plugin is still very rough around the edges, and has only been > > > tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any > > > errors. In Mike's article, he graphed segments by physical size, while my > > > plugin graphs by doc count. ES provides the physical size of a segment but > > > it doesn't specify how big the deleted\_docs section is (just number of > > > deleted docs). ES also doesn't expose which segments are merging, so I > > > can't color actively merging segments like Mike did. > > > > > > Honestly, I doubt this plugin is very practical or useful...but it is > > > fun to watch segments merge as you index/delete from your data. =) > > > -Zach > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:51am UTC](https://discuss.elastic.co/t/ann-segmentspy-site-plugin-to-watch-segments-in-realtime/10608/7 "2017-07-06T02:51:45Z")