[ANN] SegmentSpy - Site Plugin to watch segments in realtime

A while ago, Lukas showed me an article titled "Visualizing Lucene's
Segment Mergeshttp://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html",
written by Mike McCandless on his blog. Since reading that article I've
been enamored with the segment visualizations that he created to
demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index
Segments API, so I created a fairly simple site plugin to graph this data:
ElasticSearch-SegmentSpyhttps://github.com/polyfractal/elasticsearch-segmentspy

The plugin is still very rough around the edges, and has only been tested
on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In
Mike's article, he graphed segments by physical size, while my plugin
graphs by doc count. ES provides the physical size of a segment but it
doesn't specify how big the deleted_docs section is (just number of deleted
docs). ES also doesn't expose which segments are merging, so I can't color
actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun
to watch segments merge as you index/delete from your data. =)
-Zach

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Very cool!

For the deleted docs in the videos I think I just pro-rated according to
%tg deleted docs.

Mike

http://blog.mikemccandless.com

On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong zacharyjtong@gmail.com wrote:

A while ago, Lukas showed me an article titled "Visualizing Lucene's
Segment Mergeshttp://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html",
written by Mike McCandless on his blog. Since reading that article I've
been enamored with the segment visualizations that he created to
demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index
Segments API, so I created a fairly simple site plugin to graph this data:
ElasticSearch-SegmentSpyhttps://github.com/polyfractal/elasticsearch-segmentspy

The plugin is still very rough around the edges, and has only been tested
on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In
Mike's article, he graphed segments by physical size, while my plugin
graphs by doc count. ES provides the physical size of a segment but it
doesn't specify how big the deleted_docs section is (just number of deleted
docs). ES also doesn't expose which segments are merging, so I can't color
actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun
to watch segments merge as you index/delete from your data. =)
-Zach

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ahh, that makes sense. I'll make some changes when I get a chance to
account for that...I'd be happier if the segments represented size instead
of docs. =)

Thanks for taking a look at the plugin!
-Zach

On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:

Very cool!

For the deleted docs in the videos I think I just pro-rated according to
%tg deleted docs.

Mike

http://blog.mikemccandless.com

On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong <zachar...@gmail.com<javascript:>

wrote:

A while ago, Lukas showed me an article titled "Visualizing Lucene's
Segment Mergeshttp://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html",
written by Mike McCandless on his blog. Since reading that article I've
been enamored with the segment visualizations that he created to
demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the
Index Segments API, so I created a fairly simple site plugin to graph this
data: ElasticSearch-SegmentSpyhttps://github.com/polyfractal/elasticsearch-segmentspy

The plugin is still very rough around the edges, and has only been tested
on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In
Mike's article, he graphed segments by physical size, while my plugin
graphs by doc count. ES provides the physical size of a segment but it
doesn't specify how big the deleted_docs section is (just number of deleted
docs). ES also doesn't expose which segments are merging, so I can't color
actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun
to watch segments merge as you index/delete from your data. =)
-Zach

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Very nice!. We will check if we can add a flag to the segments API to indicate that its begin merged. Can you open an issue?

On Feb 4, 2013, at 1:51 PM, Zachary Tong zacharyjtong@gmail.com wrote:

Ahh, that makes sense. I'll make some changes when I get a chance to account for that...I'd be happier if the segments represented size instead of docs. =)

Thanks for taking a look at the plugin!
-Zach

On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:
Very cool!

For the deleted docs in the videos I think I just pro-rated according to %tg deleted docs.

Mike

http://blog.mikemccandless.com

On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong zachar...@gmail.com wrote:
A while ago, Lukas showed me an article titled "Visualizing Lucene's Segment Merges", written by Mike McCandless on his blog. Since reading that article I've been enamored with the segment visualizations that he created to demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the Index Segments API, so I created a fairly simple site plugin to graph this data: ElasticSearch-SegmentSpy

The plugin is still very rough around the edges, and has only been tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any errors. In Mike's article, he graphed segments by physical size, while my plugin graphs by doc count. ES provides the physical size of a segment but it doesn't specify how big the deleted_docs section is (just number of deleted docs). ES also doesn't expose which segments are merging, so I can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is fun to watch segments merge as you index/delete from your data. =)
-Zach

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I've really enjoyed this plugin, it makes watching a large Bulk load
interesting. Even with our simple 3-node, 5 shard cluster though one can't
see the 'whole picture'. This is where a 3-D grid column chart would look
very cool.

Each Index could be a grid, with extra z-axis used to display the other
shards. So X-axis is each segment as it is now, and the Y-axis is the
logarithmic size as it is now, but then plot each other shard via the
z-axis. This way one chart for each index can be seen on one page.

Obviously your JS chart library may not have that chart feature though, but
I think it would look awesome, and be more practical for even relatively
large indices?

I tried mocking this up in Excel but sadly it thinks it's too complex to
display even just 2 series with 5 rows.. Stupid Excel... ? something
looking like this anyway:
http://www.agocg.ac.uk/reports/graphics/34/appii97/image18.gif as an
example.

cheers,

Paul

On 13 February 2013 09:29, kimchy@gmail.com wrote:

Very nice!. We will check if we can add a flag to the segments API to
indicate that its begin merged. Can you open an issue?

On Feb 4, 2013, at 1:51 PM, Zachary Tong zacharyjtong@gmail.com wrote:

Ahh, that makes sense. I'll make some changes when I get a chance to
account for that...I'd be happier if the segments represented size instead
of docs. =)

Thanks for taking a look at the plugin!
-Zach

On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:

Very cool!

For the deleted docs in the videos I think I just pro-rated according to
%tg deleted docs.

Mike

http://blog.mikemccandless.com

On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong zachar...@gmail.com wrote:

A while ago, Lukas showed me an article titled "Visualizing Lucene's
Segment Mergeshttp://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html",
written by Mike McCandless on his blog. Since reading that article I've
been enamored with the segment visualizations that he created to
demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the
Index Segments API, so I created a fairly simple site plugin to graph this
data: ElasticSearch-SegmentSpyhttps://github.com/polyfractal/elasticsearch-segmentspy

The plugin is still very rough around the edges, and has only been
tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any
errors. In Mike's article, he graphed segments by physical size, while my
plugin graphs by doc count. ES provides the physical size of a segment but
it doesn't specify how big the deleted_docs section is (just number of
deleted docs). ES also doesn't expose which segments are merging, so I
can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is
fun to watch segments merge as you index/delete from your data. =)
-Zach

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Very nice!. We will check if we can add a flag to the segments API to
indicate that its begin merged. Can you open an issue?

Thanks! I'll go open an issue right now...that'd be a cool feature to
include in the plugin.

Even with our simple 3-node, 5 shard cluster though one can't see the

'whole picture'. This is where a 3-D grid column chart would look very cool.

Yeah, I totally agree. My cluster has 3 nodes as well, and it quickly
becomes unmanageable to watch all the indices. I experimented with making
all the graphs smaller (ala BigDesk), and it sorta works, but is still
difficult to get everything on screen at a reasonable resolution.

I'll play around with 3D charts, that's an excellent idea. Karmi has
convinced me to switch to D3, so more advanced visualizations should
definitely be possible.

Thanks for the comments!

-Zach

On Tuesday, February 12, 2013 5:29:40 PM UTC-5, kimchy wrote:

Very nice!. We will check if we can add a flag to the segments API to
indicate that its begin merged. Can you open an issue?

On Feb 4, 2013, at 1:51 PM, Zachary Tong <zachar...@gmail.com<javascript:>>
wrote:

Ahh, that makes sense. I'll make some changes when I get a chance to
account for that...I'd be happier if the segments represented size instead
of docs. =)

Thanks for taking a look at the plugin!
-Zach

On Sunday, February 3, 2013 7:36:46 PM UTC-5, Michael McCandless wrote:

Very cool!

For the deleted docs in the videos I think I just pro-rated according to
%tg deleted docs.

Mike

http://blog.mikemccandless.com

On Sun, Feb 3, 2013 at 9:54 AM, Zachary Tong zachar...@gmail.com wrote:

A while ago, Lukas showed me an article titled "Visualizing Lucene's
Segment Mergeshttp://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html",
written by Mike McCandless on his blog. Since reading that article I've
been enamored with the segment visualizations that he created to
demonstrate different merge policies.

ElasticSearch provides some of the necessary information through the
Index Segments API, so I created a fairly simple site plugin to graph this
data: ElasticSearch-SegmentSpyhttps://github.com/polyfractal/elasticsearch-segmentspy

The plugin is still very rough around the edges, and has only been
tested on 0.20.2 with Firefox/Chrome. Let me know if you run into any
errors. In Mike's article, he graphed segments by physical size, while my
plugin graphs by doc count. ES provides the physical size of a segment but
it doesn't specify how big the deleted_docs section is (just number of
deleted docs). ES also doesn't expose which segments are merging, so I
can't color actively merging segments like Mike did.

Honestly, I doubt this plugin is very practical or useful...but it is
fun to watch segments merge as you index/delete from your data. =)
-Zach

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.