[Ann] Elasticsearch Skywalker Plugin

Elastic Stack Elasticsearch
1

Hi,

some discussions lately - for example I
enjoyed https://github.com/elasticsearch/elasticsearch/issues/1934 with
Stefan Fussenegger - showed me that I was rather clueless about the JSON
object field model and how it maps to Lucene fields. So I started to look
if I could examine the IndexReaders, Stores, and other objects down in the
very deep internals of a shard.

My idea was to check out how Luke works and to implement some of the
interesting aspects of Luke the Elasticsearch way. The Skywalker plugin for
Elasticsearch is like Luke for Lucene.

Well, almost, it's not coming with a GUI right now.

Skywalking gives you

a deeper insight of how Elasticsearch stores Lucene indices and fields

lists timestamps and checksums of Elasticsearch's Lucene files

shows field names, both from mapping and Lucene indices

shows field types and attributes, both from mapping and Lucene indices

retrieves a ranked list of most frequent terms

Because Elasticsearch and Elasticsearch Head offers already a lot what Luke
does with Lucene, Skywalker has rather few features, just to complement
some missing parts.
More details are here: https://github.com/jprante/elasticsearch-skywalker

In the hope it's useful,

Jörg

2

That's great Jörg ! I will play with it tomorrow...
Thanks for sharing it to us.

David

Le 30 mai 2012 à 22:20, Jörg Prante joergprante@gmail.com a écrit :

Hi,

some discussions lately - for example I enjoyed Add an optional _all field like index to mapped objects · Issue #1934 · elastic/elasticsearch · GitHub with Stefan Fussenegger - showed me that I was rather clueless about the JSON object field model and how it maps to Lucene fields. So I started to look if I could examine the IndexReaders, Stores, and other objects down in the very deep internals of a shard.

My idea was to check out how Luke works and to implement some of the interesting aspects of Luke the Elasticsearch way. The Skywalker plugin for Elasticsearch is like Luke for Lucene.
Well, almost, it's not coming with a GUI right now.

Skywalking gives you

a deeper insight of how Elasticsearch stores Lucene indices and fields
lists timestamps and checksums of Elasticsearch's Lucene files
shows field names, both from mapping and Lucene indices
shows field types and attributes, both from mapping and Lucene indices
retrieves a ranked list of most frequent terms
Because Elasticsearch and Elasticsearch Head offers already a lot what Luke does with Lucene, Skywalker has rather few features, just to complement some missing parts.

More details are here: GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene

In the hope it's useful,

Jörg

3

That sounds great, thanks.

-- Tanguy

Le mercredi 30 mai 2012 22:20:09 UTC+2, Jörg Prante a écrit :

Hi,

some discussions lately - for example I enjoyed
Add an optional _all field like index to mapped objects · Issue #1934 · elastic/elasticsearch · GitHub with Stefan
Fussenegger - showed me that I was rather clueless about the JSON object
field model and how it maps to Lucene fields. So I started to look if I
could examine the IndexReaders, Stores, and other objects down in the very
deep internals of a shard.

My idea was to check out how Luke works and to implement some of the
interesting aspects of Luke the Elasticsearch way. The Skywalker plugin
for Elasticsearch is like Luke for Lucene.

Well, almost, it's not coming with a GUI right now.

Skywalking gives you

a deeper insight of how Elasticsearch stores Lucene indices and fields

lists timestamps and checksums of Elasticsearch's Lucene files

shows field names, both from mapping and Lucene indices

shows field types and attributes, both from mapping and Lucene indices

retrieves a ranked list of most frequent terms

Because Elasticsearch and Elasticsearch Head offers already a lot what
Luke does with Lucene, Skywalker has rather few features, just to
complement some missing parts.
More details are here: GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene

In the hope it's useful,

Jörg

4

Very cool! I am currently investigating why my index is quite large,
so perhaps skywalker can provide some insights.

First, installing 1.0.0 does not work since there is no zip file in Github:

Trying https://github.com/downloads/jprante/elasticsearch-skywalker/elasticsearch-skywalker-1.0.0.zip...
Trying https://github.com/jprante/elasticsearch-skywalker/zipball/v1.0.0...
Failed to install jprante/elasticsearch-skywalker/1.0.0, reason:
failed to download

There is a jar file, but not a zip.

Second, I cannot get the plugin to work after install master. Is it
because I am running ES 0.19.2?

[2012-06-01 09:41:11,940][INFO ][node ] [Mantra]
{0.19.2}[7388]: initializing ...
[2012-06-01 09:41:11,947][INFO ][plugins ] [Mantra]
loaded , sites [skywalker, head]

No handler found for uri [/_skywalker] and method [GET]

Cheers,

Ivan

On Wed, May 30, 2012 at 1:20 PM, Jörg Prante joergprante@gmail.com wrote:

Hi,

some discussions lately - for example I
enjoyed Add an optional _all field like index to mapped objects · Issue #1934 · elastic/elasticsearch · GitHub with
Stefan Fussenegger - showed me that I was rather clueless about the JSON
object field model and how it maps to Lucene fields. So I started to look if
I could examine the IndexReaders, Stores, and other objects down in the very
deep internals of a shard.

My idea was to check out how Luke works and to implement some of the
interesting aspects of Luke the Elasticsearch way. The Skywalker plugin for
Elasticsearch is like Luke for Lucene.

Well, almost, it's not coming with a GUI right now.

Skywalking gives you

a deeper insight of how Elasticsearch stores Lucene indices and fields

lists timestamps and checksums of Elasticsearch's Lucene files

shows field names, both from mapping and Lucene indices

shows field types and attributes, both from mapping and Lucene indices

retrieves a ranked list of most frequent terms

Because Elasticsearch and Elasticsearch Head offers already a lot what Luke
does with Lucene, Skywalker has rather few features, just to complement some
missing parts.

More details are here: GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene

In the hope it's useful,

Jörg

5

Hi Ivan,

I added the zip file, sorry for inconvenience.

In your plugin list, skywalker is listed as a 'site' plugin. This will not
work, it should be listed under 'loaded'.

Jörg

On Friday, June 1, 2012 6:44:24 PM UTC+2, Ivan Brusic wrote:

Very cool! I am currently investigating why my index is quite large,
so perhaps skywalker can provide some insights.

First, installing 1.0.0 does not work since there is no zip file in
Github:

Trying
https://github.com/downloads/jprante/elasticsearch-skywalker/elasticsearch-skywalker-1.0.0.zip...

Trying https://github.com/jprante/elasticsearch-skywalker/zipball/v1.0.0...

Failed to install jprante/elasticsearch-skywalker/1.0.0, reason:
failed to download

There is a jar file, but not a zip.

Second, I cannot get the plugin to work after install master. Is it
because I am running ES 0.19.2?

[2012-06-01 09:41:11,940][INFO ][node ] [Mantra]
{0.19.2}[7388]: initializing ...
[2012-06-01 09:41:11,947][INFO ][plugins ] [Mantra]
loaded , sites [skywalker, head]

No handler found for uri [/_skywalker] and method [GET]

Cheers,

Ivan

On Wed, May 30, 2012 at 1:20 PM, Jörg Prante joergprante@gmail.com
wrote:

Hi,

some discussions lately - for example I
enjoyed Add an optional _all field like index to mapped objects · Issue #1934 · elastic/elasticsearch · GitHub with
Stefan Fussenegger - showed me that I was rather clueless about the JSON
object field model and how it maps to Lucene fields. So I started to
look if
I could examine the IndexReaders, Stores, and other objects down in the
very
deep internals of a shard.

My idea was to check out how Luke works and to implement some of the
interesting aspects of Luke the Elasticsearch way. The Skywalker plugin
for
Elasticsearch is like Luke for Lucene.

Well, almost, it's not coming with a GUI right now.

Skywalking gives you

a deeper insight of how Elasticsearch stores Lucene indices and fields

lists timestamps and checksums of Elasticsearch's Lucene files

shows field names, both from mapping and Lucene indices

shows field types and attributes, both from mapping and Lucene indices

retrieves a ranked list of most frequent terms

Because Elasticsearch and Elasticsearch Head offers already a lot what
Luke
does with Lucene, Skywalker has rather few features, just to complement
some
missing parts.

More details are here:
GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene

In the hope it's useful,

Jörg

6

No problems. Just trying to help out the project by highlighting issues.

Will test the site issue next week. Should it work with ES 0.19.2?

--
Ivan

On Fri, Jun 1, 2012 at 4:09 PM, Jörg Prante joergprante@gmail.com wrote:

Hi Ivan,

I added the zip file, sorry for inconvenience.

In your plugin list, skywalker is listed as a 'site' plugin. This will not
work, it should be listed under 'loaded'.

Jörg

On Friday, June 1, 2012 6:44:24 PM UTC+2, Ivan Brusic wrote:

Very cool! I am currently investigating why my index is quite large,
so perhaps skywalker can provide some insights.

First, installing 1.0.0 does not work since there is no zip file in
Github:

Trying
https://github.com/downloads/jprante/elasticsearch-skywalker/elasticsearch-skywalker-1.0.0.zip...
Trying
https://github.com/jprante/elasticsearch-skywalker/zipball/v1.0.0...
Failed to install jprante/elasticsearch-skywalker/1.0.0, reason:
failed to download

There is a jar file, but not a zip.

Second, I cannot get the plugin to work after install master. Is it
because I am running ES 0.19.2?

[2012-06-01 09:41:11,940][INFO ][node ] [Mantra]
{0.19.2}[7388]: initializing ...
[2012-06-01 09:41:11,947][INFO ][plugins ] [Mantra]
loaded , sites [skywalker, head]

No handler found for uri [/_skywalker] and method [GET]

Cheers,

Ivan

On Wed, May 30, 2012 at 1:20 PM, Jörg Prante joergprante@gmail.com
wrote:

Hi,

some discussions lately - for example I
enjoyed Add an optional _all field like index to mapped objects · Issue #1934 · elastic/elasticsearch · GitHub with
Stefan Fussenegger - showed me that I was rather clueless about the JSON
object field model and how it maps to Lucene fields. So I started to
look if
I could examine the IndexReaders, Stores, and other objects down in the
very
deep internals of a shard.

My idea was to check out how Luke works and to implement some of the
interesting aspects of Luke the Elasticsearch way. The Skywalker plugin
for
Elasticsearch is like Luke for Lucene.

Well, almost, it's not coming with a GUI right now.

Skywalking gives you

a deeper insight of how Elasticsearch stores Lucene indices and fields

lists timestamps and checksums of Elasticsearch's Lucene files

shows field names, both from mapping and Lucene indices

shows field types and attributes, both from mapping and Lucene indices

retrieves a ranked list of most frequent terms

Because Elasticsearch and Elasticsearch Head offers already a lot what
Luke
does with Lucene, Skywalker has rather few features, just to complement
some
missing parts.

More details are here:
GitHub - jprante/elasticsearch-skywalker: Skywalker for Elasticsearch is like Luke for Lucene

In the hope it's useful,

Jörg