Search the http://elasticsearch.org website with elasticsearch itself

Elastic Stack Elasticsearch
1

Hi,

I've spent couple of hours during the last days with implementing an ElasticSearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can try any Lucene-compatible query.

The code for the demo is available here: https://github.com/karmi/elasticsearch.github.com/compare/elasticsearch%3Amaster...search. The autocomplete + results display is done with heavily stylized jQuery-UI Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it, though.

The content was insterted into ES with this Ruby library: https://github.com/karmi/hide, it leverages the Jekyll gem to gather info about the pages, so the usage is trivial. It uses the https://github.com/karmi/slingshot gem to insert docs into ES (see https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new content from Github whenever there's a push into the repo, and re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow search in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any ideas, suggestions?

Karel

2

+1

As a suggestion, I would expect a list of results when clicking ENTER on the
search box, instead of direct navigation to the first result.

On Fri, Feb 11, 2011 at 3:52 PM, Karel Minařík karel.minarik@gmail.comwrote:

Hi,

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can
try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsearch%3Amaster...search.
The autocomplete + results display is done with heavily stylized jQuery-UI
Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it,
though.

The content was insterted into ES with this Ruby library:
GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses the
GitHub - karmi/slingshot: Ruby client for ElasticSearch gem to insert docs into ES (see
https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the
    least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new
    content from Github whenever there's a push into the repo, and re-indexes
    accordingly.

For the optimal case, it would be nice to have checkboxes to allow search
in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it
would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any
ideas, suggestions?

Karel

3

Karel,

Thank you for the work.

I am curious though, when I search for "https" on the site, the

first result is this page:

http://elasticsearch.karmi.cz/guide/appendix/clients.html

And yet, searching the contents of the page, https is nowhere to be

found.

How exactly was content indexed that it would give me that result?

	- Nick

-----Original Message-----
From: Karel Minařík [mailto:karel.minarik@gmail.com]
Sent: Friday, February 11, 2011 9:53 AM
To: users@elasticsearch.com
Subject: Search the http://elasticsearch.org website with elasticsearch
itself

Hi,

I've spent couple of hours during the last days with implementing an
ElasticSearch-backed search for the ES website (so it could be
"self-hosted", so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can
try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsearch%3Ama
ster...search. The autocomplete + results display is done with heavily
stylized jQuery-UI Autocomplete. It's kinda invasive, but works. Would be
nice to rewrite it, though.

The content was insterted into ES with this Ruby library:
https://github.com/karmi/hide, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses the
https://github.com/karmi/slingshot gem to insert docs into ES (see
https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the
    least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new
    content from Github whenever there's a push into the repo, and re-indexes
    accordingly.

For the optimal case, it would be nice to have checkboxes to allow search in
"guides" etc only, @lukasvlcek suggested indexing also Github issues, it
would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any ideas,
suggestions?

Karel

4

Cool!

On 11 Feb., 15:52, Karel Minařík karel.mina...@gmail.com wrote:

Hi,

I've spent couple of hours during the last days with implementing an Elasticsearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

First, you can try the functionality here:

-->http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can try any Lucene-compatible query.

The code for the demo is available here:https://github.com/karmi/elasticsearch.github.com/compare/elasticsear.... The autocomplete + results display is done with heavily stylized jQuery-UI Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it, though.

The content was insterted into ES with this Ruby library:GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info about the pages, so the usage is trivial. It uses thehttps://github.com/karmi/slingshotgem to insert docs into ES (seehttps://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new content from Github whenever there's a push into the repo, and re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow search in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any ideas, suggestions?

Karel

5

Hi Karmi

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC
the other day.

Looking great!

What would also be good is to index:

  • the github issues
  • the forum

There is loads of useful advice in both of those resources, and the
current search functionality is a bit poor.

clint

6

Really cool, way to go @karmi!
Actually, I did not suggest searching the GitHub issues (it must have been
someone else) but it is good idea. May be you saw me complaining about
GitHub search (and you deduced that I wanted to improve it which is not the
same but I do not object). :slight_smile:

+1

Lukas

On Fri, Feb 11, 2011 at 3:52 PM, Karel Minařík karel.minarik@gmail.comwrote:

Hi,

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can
try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsearch%3Amaster...search.
The autocomplete + results display is done with heavily stylized jQuery-UI
Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it,
though.

The content was insterted into ES with this Ruby library:
GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses the
GitHub - karmi/slingshot: Ruby client for ElasticSearch gem to insert docs into ES (see
https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the
    least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new
    content from Github whenever there's a push into the repo, and re-indexes
    accordingly.

For the optimal case, it would be nice to have checkboxes to allow search
in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it
would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any
ideas, suggestions?

Karel

7

Hi all,

thanks for the feedback!

@Enrique: Sure, but bear in mind the logic is pure JavaScript, so the
website could be hosted at Github with no frills. The interactivity
needs some polishing badly, for sure.

@Nick: No idea about the "https", the content is taken from the Jekyll
page [see here: https://github.com/karmi/hide/blob/master/lib/hide/page.rb#L18-20].
As I mentioned, setting up a proper mapping is badly needed for the
thing to work. At the moment, I just shoved the JSONs into ES.

@Clint: Yes, definitely, indexing Issues and Forum would be naturall
addition, and in fact very simple to do. However, there would have to
be an easy way to limit the search to document type -- I'm not sure if
by adding some checkboxes (and thus polluting the GUI) or (rather)
something like: q=mapping category:issues OR category:forum. The
idea is that we could use the searchbox as a way to teach people
search tricks (and save them into cookies, etc etc etc). That said,
I'd very much like to build some sort of "search tips" into it.

@Lukas: Github Issues suck badly from my POV :), that being said while
I love Github and consider it one of the most (if not the most)
important things to happen to open-source in the last decade.

Karel

On Feb 11, 4:33 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Really cool, way to go @karmi!
Actually, I did not suggest searching the GitHub issues (it must have been
someone else) but it is good idea. May be you saw me complaining about
GitHub search (and you deduced that I wanted to improve it which is not the
same but I do not object). :slight_smile:

+1

Lukas

On Fri, Feb 11, 2011 at 3:52 PM, Karel Minařík karel.mina...@gmail.comwrote:

Hi,

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

-->http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can
try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsear....
The autocomplete + results display is done with heavily stylized jQuery-UI
Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it,
though.

The content was insterted into ES with this Ruby library:
GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses the
https://github.com/karmi/slingshotgem to insert docs into ES (see
https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the
    least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new
    content from Github whenever there's a push into the repo, and re-indexes
    accordingly.

For the optimal case, it would be nice to have checkboxes to allow search
in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it
would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any
ideas, suggestions?

Karel

8

Heya,

Karel, great job!. It would be great to add this to the main site. The harder parts, I guess is to hook up to changes from github.

Also, something that would be really cool is to have all of this provided to any github project. It can be (one) of the killer applications built on top of ES. Think of a site (github.elasticsearch.org?), where people can add their username to, and it will automatically provide proper search features for the issues, pages power site, and possibly also code repositories. That can be nice :). Of course, this will become less value once github will fix the way they provide search on their site :).

-shay.banon
On Friday, February 11, 2011 at 7:27 PM, Karel Minarik wrote:

Hi all,

thanks for the feedback!

@Enrique: Sure, but bear in mind the logic is pure JavaScript, so the
website could be hosted at Github with no frills. The interactivity
needs some polishing badly, for sure.

@Nick: No idea about the "https", the content is taken from the Jekyll
page [see here: https://github.com/karmi/hide/blob/master/lib/hide/page.rb#L18-20].
As I mentioned, setting up a proper mapping is badly needed for the
thing to work. At the moment, I just shoved the JSONs into ES.

@Clint: Yes, definitely, indexing Issues and Forum would be naturall
addition, and in fact very simple to do. However, there would have to
be an easy way to limit the search to document type -- I'm not sure if
by adding some checkboxes (and thus polluting the GUI) or (rather)
something like: q=mapping category:issues OR category:forum. The
idea is that we could use the searchbox as a way to teach people
search tricks (and save them into cookies, etc etc etc). That said,
I'd very much like to build some sort of "search tips" into it.

@Lukas: Github Issues suck badly from my POV :), that being said while
I love Github and consider it one of the most (if not the most)
important things to happen to open-source in the last decade.

Karel

On Feb 11, 4:33 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Really cool, way to go @karmi!
Actually, I did not suggest searching the GitHub issues (it must have been
someone else) but it is good idea. May be you saw me complaining about
GitHub search (and you deduced that I wanted to improve it which is not the
same but I do not object). :slight_smile:

+1

Lukas

On Fri, Feb 11, 2011 at 3:52 PM, Karel Minařík karel.mina...@gmail.comwrote:

Hi,

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

-->http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can
try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsear....
The autocomplete + results display is done with heavily stylized jQuery-UI
Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it,
though.

The content was insterted into ES with this Ruby library:
GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses the
https://github.com/karmi/slingshotgem to insert docs into ES (see
https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the
    least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new
    content from Github whenever there's a push into the repo, and re-indexes
    accordingly.

For the optimal case, it would be nice to have checkboxes to allow search
in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it
would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any
ideas, suggestions?

Karel

9

This is really, really cool!
And very helpful too, as the google index is really confused right now
since the site restructuring.

Thanks!

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an Elasticsearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

10

Worked on getting the google index better by changing the format of the site a bit. Hopefully it will be better now,
On Wednesday, February 16, 2011 at 1:49 PM, Kristian Jörg wrote:

This is really, really cool!
And very helpful too, as the google index is really confused right now
since the site restructuring.

Thanks!

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an Elasticsearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

11

Hi Karel,

since 0.15-release yoursearch enginesite is now out of date. Any hopes
of getting it updated?

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an Elasticsearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can try any Lucene-compatible query.

The code for the demo is available here: https://github.com/karmi/elasticsearch.github.com/compare/elasticsearch%3Amaster...search. The autocomplete + results display is done with heavily stylized jQuery-UI Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it, though.

The content was insterted into ES with this Ruby library: GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info about the pages, so the usage is trivial. It uses the GitHub - karmi/slingshot: Ruby client for ElasticSearch gem to insert docs into ES (see https://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new content from Github whenever there's a push into the repo, and re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow search in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any ideas, suggestions?

Karel

12

Hi,

I've updated the site content with current master (see eg.
http://elasticsearch.karmi.cz:9200/elastic-search-website/guide/_search?q="clear%20cache"),
there is now some accelerated effort to roll out the search feature on
the main site soon, see ticket: https://github.com/elasticsearch/elasticsearch.github.com/issues/#issue/14

Best!,

Karel

On Feb 22, 1:19 pm, Kristian Jörg k...@devo.se wrote:

Hi Karel,

since 0.15-release yoursearch enginesite is now out of date. Any hopes
of getting it updated?

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an Elasticsearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can try any Lucene-compatible query.

The code for the demo is available here:https://github.com/karmi/elasticsearch.github.com/compare/elasticsear.... The autocomplete + results display is done with heavily stylized jQuery-UI Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it, though.

The content was insterted into ES with this Ruby library:GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info about the pages, so the usage is trivial. It uses thehttps://github.com/karmi/slingshotgem to insert docs into ES (seehttps://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new content from Github whenever there's a push into the repo, and re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow search in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any ideas, suggestions?

Karel

13

Cool, it is great to see this coming!

On Wed, Mar 2, 2011 at 4:00 PM, Karel Minarik karel.minarik@gmail.comwrote:

Hi,

I've updated the site content with current master (see eg.

http://elasticsearch.karmi.cz:9200/elastic-search-website/guide/_search?q="clear%20cache"
),
there is now some accelerated effort to roll out the search feature on
the main site soon, see ticket:
https://github.com/elasticsearch/elasticsearch.github.com/issues/#issue/14

Best!,

Karel

On Feb 22, 1:19 pm, Kristian Jörg k...@devo.se wrote:

Hi Karel,

since 0.15-release yoursearch enginesite is now out of date. Any hopes
of getting it updated?

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you
can try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsear....
The autocomplete + results display is done with heavily stylized jQuery-UI
Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it,
though.

The content was insterted into ES with this Ruby library:
GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses thehttps://
github.com/karmi/slingshotgem to insert docs into ES (seehttps://
github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is
    the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls
    new content from Github whenever there's a push into the repo, and
    re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow
search in "guides" etc only, @lukasvlcek suggested indexing also Github
issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any
ideas, suggestions?

Karel

14

Yep ! I cant wait to have a searchable documentation.

On Wed, Mar 2, 2011 at 4:17 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Cool, it is great to see this coming!

On Wed, Mar 2, 2011 at 4:00 PM, Karel Minarik karel.minarik@gmail.comwrote:

Hi,

I've updated the site content with current master (see eg.

http://elasticsearch.karmi.cz:9200/elastic-search-website/guide/_search?q="clear%20cache"
),
there is now some accelerated effort to roll out the search feature on
the main site soon, see ticket:
https://github.com/elasticsearch/elasticsearch.github.com/issues/#issue/14

Best!,

Karel

On Feb 22, 1:19 pm, Kristian Jörg k...@devo.se wrote:

Hi Karel,

since 0.15-release yoursearch enginesite is now out of date. Any hopes
of getting it updated?

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an
Elasticsearch-backed search for the ES website (so it could be
“self-hosted”, so to speak), as we have been talking about it on IRC the
other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you
can try any Lucene-compatible query.

The code for the demo is available here:
https://github.com/karmi/elasticsearch.github.com/compare/elasticsear....
The autocomplete + results display is done with heavily stylized jQuery-UI
Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it,
though.

The content was insterted into ES with this Ruby library:
GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info
about the pages, so the usage is trivial. It uses thehttps://
github.com/karmi/slingshotgem to insert docs into ES (seehttps://
github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is
    the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different
    categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls
    new content from Github whenever there's a push into the repo, and
    re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow
search in "guides" etc only, @lukasvlcek suggested indexing also Github
issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any
ideas, suggestions?

Karel

--
Alexandre Heimburger
R&D Manager
blueKiwi Software
tel : +33687880997
email : ahb@bluekiwi-software.com
adress : 93 rue Vieille du Temple, 75003 Paris

What is blueKiwi? blueKiwi - the first Enterprise Social Software Suite in
the world building professional networks on conversations and relationships

  • helps large organizations increase their productivity, foster innovations
    and boost people satisfaction.
15

Great work karmi, thanks for all your effort!, thats what open source and community is all about.
On Wednesday, March 2, 2011 at 6:05 PM, Alexandre Heimburger wrote:

Yep ! I cant wait to have a searchable documentation.

On Wed, Mar 2, 2011 at 4:17 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Cool, it is great to see this coming!

On Wed, Mar 2, 2011 at 4:00 PM, Karel Minarik karel.minarik@gmail.com wrote:

Hi,

I've updated the site content with current master (see eg.
http://elasticsearch.karmi.cz:9200/elastic-search-website/guide/_search?q="clear%20cache"),
there is now some accelerated effort to roll out the search feature on
the main site soon, see ticket: https://github.com/elasticsearch/elasticsearch.github.com/issues/#issue/14

Best!,

Karel

On Feb 22, 1:19 pm, Kristian Jörg k...@devo.se wrote:

Hi Karel,

since 0.15-release yoursearch enginesite is now out of date. Any hopes
of getting it updated?

/Kristian

Karel Minařík skrev 2011-02-11 15:52:

Hi,

I've spent couple of hours during the last days with implementing an Elasticsearch-backed search for the ES website (so it could be “self-hosted”, so to speak), as we have been talking about it on IRC the other day.

First, you can try the functionality here:

--> http://elasticsearch.karmi.cz/

The query is passed directly as a query string parameter to ES, so you can try any Lucene-compatible query.

The code for the demo is available here:https://github.com/karmi/elasticsearch.github.com/compare/elasticsear.... The autocomplete + results display is done with heavily stylized jQuery-UI Autocomplete. It's kinda invasive, but works. Would be nice to rewrite it, though.

The content was insterted into ES with this Ruby library:GitHub - karmi/hide: Search for Jekyll websites, it leverages the Jekyll gem to gather info about the pages, so the usage is trivial. It uses thehttps://github.com/karmi/slingshotgem to insert docs into ES (seehttps://github.com/karmi/hide/blob/master/lib/hide/indexer.rb).

For the real use, I think the following would need to be done:

  • Get hold of some VPS or other server to run the ES server. This is the least of a problem, either me or @clintongormley can provide one.
  • A redesign of the results window, color-differentiate the different categories (blog, guide)
  • Displaying a highlight of term instead of URL
  • Creating an HTTP endpoint for Github post-receive hook, which pulls new content from Github whenever there's a push into the repo, and re-indexes accordingly.

For the optimal case, it would be nice to have checkboxes to allow search in "guides" etc only, @lukasvlcek suggested indexing also Github issues, it would be nice to have prettier GUI, etc etc etc.

What does everybody think, is it worthwile to continue with this? Any ideas, suggestions?

Karel

--
Alexandre Heimburger
R&D Manager
blueKiwi Software
tel : +33687880997
email : ahb@bluekiwi-software.com
adress : 93 rue Vieille du Temple, 75003 Paris

What is blueKiwi? blueKiwi - the first Enterprise Social Software Suite in the world building professional networks on conversations and relationships - helps large organizations increase their productivity, foster innovations and boost people satisfaction.

16

Hi all,

just to inform you, that we are almost ready to migrate the search to
the main site. We have the ES server with index, infrastructure for
continuous index updating based on Github post-receive hooks, and the
UI is "good" for the time being.

You can try out the searches:

--> http://search.elasticsearch.org/_search?q=filter&fields=title

Note, that access is limited to GET requests to the _search endpoint.

The demo (still) available at http://elasticsearch.karmi.cz/ now
points to this server as well.

As soon as we sync the ES server with the main repo and the UI stuff
[https://github.com/karmi/elasticsearch.github.com/compare/search] is
merged, it's ready to go live.

Detailed notes about the setup are here: https://gist.github.com/910301

We know about couple of mishaps like missing ulimit, we'll solve
those. If you'll see anything incorrect about this all, please send a
note.

Thanks!,

Karel

17

Great work!. Pushed the update to the site, thanks!. Just ping me with setting up the hook on github.
On Sunday, April 10, 2011 at 4:59 PM, Karel Minarik wrote:

Hi all,

just to inform you, that we are almost ready to migrate the search to
the main site. We have the ES server with index, infrastructure for
continuous index updating based on Github post-receive hooks, and the
UI is "good" for the time being.

You can try out the searches:

--> http://search.elasticsearch.org/_search?q=filter&fields=title

Note, that access is limited to GET requests to the _search endpoint.

The demo (still) available at http://elasticsearch.karmi.cz/ now
points to this server as well.

As soon as we sync the ES server with the main repo and the UI stuff
[Comparing rhec:master...karmi:search · rhec/elasticsearch.github.com · GitHub] is
merged, it's ready to go live.

Detailed notes about the setup are here: 910301’s gists · GitHub

We know about couple of mishaps like missing ulimit, we'll solve
those. If you'll see anything incorrect about this all, please send a
note.

Thanks!,

Karel

18

Cool~

From: Shay Banon
Sent: Sunday, April 10, 2011 10:51 PM
To: users@elasticsearch.com
Subject: Re: Search the http://elasticsearch.org website with elasticsearch itself

Great work!. Pushed the update to the site, thanks!. Just ping me with setting up the hook on github.

On Sunday, April 10, 2011 at 4:59 PM, Karel Minarik wrote:

Hi all,

just to inform you, that we are almost ready to migrate the search to
the main site. We have the ES server with index, infrastructure for
continuous index updating based on Github post-receive hooks, and the
UI is "good" for the time being.

You can try out the searches:

--> http://search.elasticsearch.org/_search?q=filter&fields=title

Note, that access is limited to GET requests to the _search endpoint.

The demo (still) available at http://elasticsearch.karmi.cz/ now
points to this server as well.

As soon as we sync the ES server with the main repo and the UI stuff
[https://github.com/karmi/elasticsearch.github.com/compare/search] is
merged, it's ready to go live.

Detailed notes about the setup are here: https://gist.github.com/910301

We know about couple of mishaps like missing ulimit, we'll solve
those. If you'll see anything incorrect about this all, please send a
note.

Thanks!,

Karel

19

Thanks for this helpful stuff!!!

On Sun, Apr 10, 2011 at 3:59 PM, Karel Minarik karel.minarik@gmail.comwrote:

Hi all,

just to inform you, that we are almost ready to migrate the search to
the main site. We have the ES server with index, infrastructure for
continuous index updating based on Github post-receive hooks, and the
UI is "good" for the time being.

You can try out the searches:

--> http://search.elasticsearch.org/_search?q=filter&fields=title

Note, that access is limited to GET requests to the _search endpoint.

The demo (still) available at http://elasticsearch.karmi.cz/ now
points to this server as well.

As soon as we sync the ES server with the main repo and the UI stuff
[Comparing rhec:master...karmi:search · rhec/elasticsearch.github.com · GitHub] is
merged, it's ready to go live.

Detailed notes about the setup are here: 910301’s gists · GitHub

We know about couple of mishaps like missing ulimit, we'll solve
those. If you'll see anything incorrect about this all, please send a
note.

Thanks!,

Karel

20

Karmi for president!
Dne 10.4.2011 15:59 "Karel Minarik" karel.minarik@gmail.com napsal(a):

Hi all,

just to inform you, that we are almost ready to migrate the search to
the main site. We have the ES server with index, infrastructure for
continuous index updating based on Github post-receive hooks, and the
UI is "good" for the time being.

You can try out the searches:

--> http://search.elasticsearch.org/_search?q=filter&fields=title

Note, that access is limited to GET requests to the _search endpoint.

The demo (still) available at http://elasticsearch.karmi.cz/ now
points to this server as well.

As soon as we sync the ES server with the main repo and the UI stuff
[Comparing rhec:master...karmi:search · rhec/elasticsearch.github.com · GitHub] is
merged, it's ready to go live.

Detailed notes about the setup are here: 910301’s gists · GitHub

We know about couple of mishaps like missing ulimit, we'll solve
those. If you'll see anything incorrect about this all, please send a
note.

Thanks!,

Karel