Getting Started

Elastic Stack Elasticsearch
21

Yes thats a great idea. I've been digging into Search the last week for the
first time and found that getting all the pieces and options together is the
most challenging part. A query DSL format would be very helpful!

From: Philippe Laflamme [mailto:philippe.laflamme@obiba.org]
Sent: vrijdag 20 mei 2011 22:42
To: users@elasticsearch.com
Subject: Re: Getting Started

What I find is missing most in the docs is a top-down description of the
Query DSL: what is the possible structure of a search request body. It's all
in the docs, but split into many little pieces which makes the "vertical"
view (as Shay puts it) difficult to grasp. This is even complicated further
by the fact that some pieces can go in multiple places (a query within a
filter for example).

What would be very useful is a formal description of the Query DSL (similar
to what an XSD offers to XML). No, not that complicated, but it would be
complete (covers all possibilities) and would show what is required and what
is optional for each "node" type. And, while I'm dreaming, it would also be
clickable so you could drill-down.

The facet documentation has a formal description (uses and
such) but is incomplete (each example only shows part of what is available):
http://www.elasticsearch.org/guide/reference/api/search/facets/

I'm not too familiar with the code that parses the queries, but could such a
document be generated from the code? Maybe using annotations or
introspection or whatever, that would make it easily maintainable...

I'd like to contribute if that makes any sense to anyone.

Philippe

On Fri, May 20, 2011 at 16:21, Shay Banon shay.banon@elasticsearch.com
wrote:

What we can do is start a section in the site with samples. I prefer not to
have several places of docs around for elasticsearch. And the site is,
effectively, a wiki (anybody can edit and send pull requests).

On Friday, May 20, 2011 at 11:19 PM, Karussell wrote:

I completely agree with that :slight_smile:

but for the sake of just working examples (sometimes user needs a
lot!) a wiki would be very nice - even if sprinkled here and there.

I started a wiki here:

http://elastic.wikispaces.com

22

2011/5/20 Shay Banon shay.banon@elasticsearch.com

Not going to stand in the way of a wiki, but I am certainly not going to
spend any time there. There is enough things to do already. I hope there
won't be one since it will create confusion for users as wikis tend to
quickly get out of date, and its not maintained in the way that the guide
is.

If someone can't figure out how to run the github site, then its fine by me
that he won't contribute docs.

Hmmh, that's an ignorant statement IMO. I don't think that you can

  1. judge about people who have (version) problems with ruby stuff or their
    ruby OS integration like me. Or are my explanations on the mailing list
    really that stupid that you don't want me to contribute?

and 2. most documentation in many software products are NOT written by the
developers who assume too many thinks a newbie cannot know.

23

Here's what I'd really love to have:

It's probably invalid, but anyway, it gets the idea across...

https://gist.github.com/983779I tried to use the RFC-way for formal
descriptions. This syntax is used to describe HTTP headers, network
protocols and such, so it needs to be complete and being an RFC, it's very
terse. It would need to be adapted to the fact that it's JSON (the colons
and the brackets are conflicting). Now that I think about it, JSON has a
formal description on their site. Maybe it could use that syntax (re-using
the JSON terms even): www.json.org

Philippe

On Fri, May 20, 2011 at 16:55, Shay Banon shay.banon@elasticsearch.comwrote:

There is no way to generate it automatically, but contributions are
warmly welcomed.

On Friday, May 20, 2011 at 11:42 PM, Philippe Laflamme wrote:

What I find is missing most in the docs is a top-down description of the
Query DSL: what is the possible structure of a search request body. It's all
in the docs, but split into many little pieces which makes the "vertical"
view (as Shay puts it) difficult to grasp. This is even complicated further
by the fact that some pieces can go in multiple places (a query within a
filter for example).

What would be very useful is a formal description of the Query DSL (similar
to what an XSD offers to XML). No, not that complicated, but it would be
complete (covers all possibilities) and would show what is required and what
is optional for each "node" type. And, while I'm dreaming, it would also be
clickable so you could drill-down.

The facet documentation has a formal description (uses and
such) but is incomplete (each example only shows part of what is
available):
Elasticsearch Platform — Find real-time answers at scale | Elastic

I'm not too familiar with the code that parses the queries, but could such
a document be generated from the code? Maybe using annotations or
introspection or whatever, that would make it easily maintainable...

I'd like to contribute if that makes any sense to anyone.

Philippe

On Fri, May 20, 2011 at 16:21, Shay Banon shay.banon@elasticsearch.comwrote:

What we can do is start a section in the site with samples. I prefer not to
have several places of docs around for elasticsearch. And the site is,
effectively, a wiki (anybody can edit and send pull requests).

On Friday, May 20, 2011 at 11:19 PM, Karussell wrote:

I completely agree with that :slight_smile:

but for the sake of just working examples (sometimes user needs a
lot!) a wiki would be very nice - even if sprinkled here and there.

I started a wiki here:

http://elastic.wikispaces.com

24

Lol, after complaining I tried to install it again with success :slight_smile:

But my system previously didn't use rvm which is a great simplifier for at
least my crappy (?) system ...

2011/5/20 Stefan Müller tableyourtime@googlemail.com

2011/5/20 Shay Banon shay.banon@elasticsearch.com

Not going to stand in the way of a wiki, but I am certainly not going to
spend any time there. There is enough things to do already. I hope there
won't be one since it will create confusion for users as wikis tend to
quickly get out of date, and its not maintained in the way that the guide
is.

If someone can't figure out how to run the github site, then its fine by
me that he won't contribute docs.

Hmmh, that's an ignorant statement IMO. I don't think that you can

  1. judge about people who have (version) problems with ruby stuff or their
    ruby OS integration like me. Or are my explanations on the mailing list
    really that stupid that you don't want me to contribute?

and 2. most documentation in many software products are NOT written by the
developers who assume too many thinks a newbie cannot know.

25

On Saturday, May 21, 2011 at 12:03 AM, Stefan Müller wrote:

2011/5/20 Shay Banon shay.banon@elasticsearch.com

Not going to stand in the way of a wiki, but I am certainly not going to spend any time there. There is enough things to do already. I hope there won't be one since it will create confusion for users as wikis tend to quickly get out of date, and its not maintained in the way that the guide is.

If someone can't figure out how to run the github site, then its fine by me that he won't contribute docs.
Hmmh, that's an ignorant statement IMO. I don't think that you can

  1. judge about people who have (version) problems with ruby stuff or their ruby OS integration like me. Or are my explanations on the mailing list really that stupid that you don't want me to contribute?
    Small changes don't require you to know how to run the site, since its just textile files. Github even allow you to edit in place and send pull requests all within the browser.

Bigger changes do require you to know how to do that, and yes, if you are going to become a major contributor to the site / guide, then you will have to find out how to run it. And if you can't figure out how to run the site, and thats whats stopping you from contributing docs, then I am fine that you won't contribute. No judging.

Have no idea what you are talking about in the mailing list.

and 2. most documentation in many software products are NOT written by the developers who assume too many thinks a newbie cannot know.
I strongly disagree here. I think the best documented open source projects out there are the ones where the people who document are also great developers. It might not be me, so here's to hoping we will have more of those in the community.

26

My gut feeling thinks that this will confuse people even more.
On Saturday, May 21, 2011 at 12:03 AM, Philippe Laflamme wrote:

Here's what I'd really love to have:

An example formal description of facets in ES · GitHub

It's probably invalid, but anyway, it gets the idea across...

I tried to use the RFC-way for formal descriptions. This syntax is used to describe HTTP headers, network protocols and such, so it needs to be complete and being an RFC, it's very terse. It would need to be adapted to the fact that it's JSON (the colons and the brackets are conflicting). Now that I think about it, JSON has a formal description on their site. Maybe it could use that syntax (re-using the JSON terms even): www.json.org

Philippe

On Fri, May 20, 2011 at 16:55, Shay Banon shay.banon@elasticsearch.com wrote:

There is no way to generate it automatically, but contributions are warmly welcomed.
On Friday, May 20, 2011 at 11:42 PM, Philippe Laflamme wrote:

What I find is missing most in the docs is a top-down description of the Query DSL: what is the possible structure of a search request body. It's all in the docs, but split into many little pieces which makes the "vertical" view (as Shay puts it) difficult to grasp. This is even complicated further by the fact that some pieces can go in multiple places (a query within a filter for example).

What would be very useful is a formal description of the Query DSL (similar to what an XSD offers to XML). No, not that complicated, but it would be complete (covers all possibilities) and would show what is required and what is optional for each "node" type. And, while I'm dreaming, it would also be clickable so you could drill-down.

The facet documentation has a formal description (uses and such) but is incomplete (each example only shows part of what is available): Elasticsearch Platform — Find real-time answers at scale | Elastic

I'm not too familiar with the code that parses the queries, but could such a document be generated from the code? Maybe using annotations or introspection or whatever, that would make it easily maintainable...

I'd like to contribute if that makes any sense to anyone.

Philippe

On Fri, May 20, 2011 at 16:21, Shay Banon shay.banon@elasticsearch.com wrote:

What we can do is start a section in the site with samples. I prefer not to have several places of docs around for elasticsearch. And the site is, effectively, a wiki (anybody can edit and send pull requests).
On Friday, May 20, 2011 at 11:19 PM, Karussell wrote:

I completely agree with that :slight_smile:

but for the sake of just working examples (sometimes user needs a
lot!) a wiki would be very nice - even if sprinkled here and there.

I started a wiki here:

http://elastic.wikispaces.com

27

I'm not sure how a formal description can be confusing since it leaves no
place for any guesswork or interpretation. Would you accept pull requests
for contributions towards making such a formal description?

Cheers,
Philippe

On Fri, May 20, 2011 at 17:35, Shay Banon shay.banon@elasticsearch.comwrote:

My gut feeling thinks that this will confuse people even more.

On Saturday, May 21, 2011 at 12:03 AM, Philippe Laflamme wrote:

Here's what I'd really love to have:

An example formal description of facets in ES · GitHub

It's probably invalid, but anyway, it gets the idea across...

https://gist.github.com/983779I tried to use the RFC-way for formal
descriptions. This syntax is used to describe HTTP headers, network
protocols and such, so it needs to be complete and being an RFC, it's very
terse. It would need to be adapted to the fact that it's JSON (the colons
and the brackets are conflicting). Now that I think about it, JSON has a
formal description on their site. Maybe it could use that syntax (re-using
the JSON terms even): www.json.org

Philippe

On Fri, May 20, 2011 at 16:55, Shay Banon shay.banon@elasticsearch.comwrote:

There is no way to generate it automatically, but contributions are
warmly welcomed.

On Friday, May 20, 2011 at 11:42 PM, Philippe Laflamme wrote:

What I find is missing most in the docs is a top-down description of the
Query DSL: what is the possible structure of a search request body. It's all
in the docs, but split into many little pieces which makes the "vertical"
view (as Shay puts it) difficult to grasp. This is even complicated further
by the fact that some pieces can go in multiple places (a query within a
filter for example).

What would be very useful is a formal description of the Query DSL (similar
to what an XSD offers to XML). No, not that complicated, but it would be
complete (covers all possibilities) and would show what is required and what
is optional for each "node" type. And, while I'm dreaming, it would also be
clickable so you could drill-down.

The facet documentation has a formal description (uses and
such) but is incomplete (each example only shows part of what is
available):
Elasticsearch Platform — Find real-time answers at scale | Elastic

I'm not too familiar with the code that parses the queries, but could such
a document be generated from the code? Maybe using annotations or
introspection or whatever, that would make it easily maintainable...

I'd like to contribute if that makes any sense to anyone.

Philippe

On Fri, May 20, 2011 at 16:21, Shay Banon shay.banon@elasticsearch.comwrote:

What we can do is start a section in the site with samples. I prefer not to
have several places of docs around for elasticsearch. And the site is,
effectively, a wiki (anybody can edit and send pull requests).

On Friday, May 20, 2011 at 11:19 PM, Karussell wrote:

I completely agree with that :slight_smile:

but for the sake of just working examples (sometimes user needs a
lot!) a wiki would be very nice - even if sprinkled here and there.

I started a wiki here:

http://elastic.wikispaces.com

28

Sure, no problem. I think ti will be confusing since not everybody know how to read it.
On Saturday, May 21, 2011 at 3:32 AM, Philippe Laflamme wrote:

I'm not sure how a formal description can be confusing since it leaves no place for any guesswork or interpretation. Would you accept pull requests for contributions towards making such a formal description?

Cheers,
Philippe

On Fri, May 20, 2011 at 17:35, Shay Banon shay.banon@elasticsearch.com wrote:

My gut feeling thinks that this will confuse people even more.
On Saturday, May 21, 2011 at 12:03 AM, Philippe Laflamme wrote:

Here's what I'd really love to have:

An example formal description of facets in ES · GitHub

It's probably invalid, but anyway, it gets the idea across...

I tried to use the RFC-way for formal descriptions. This syntax is used to describe HTTP headers, network protocols and such, so it needs to be complete and being an RFC, it's very terse. It would need to be adapted to the fact that it's JSON (the colons and the brackets are conflicting). Now that I think about it, JSON has a formal description on their site. Maybe it could use that syntax (re-using the JSON terms even): www.json.org

Philippe

On Fri, May 20, 2011 at 16:55, Shay Banon shay.banon@elasticsearch.com wrote:

There is no way to generate it automatically, but contributions are warmly welcomed.
On Friday, May 20, 2011 at 11:42 PM, Philippe Laflamme wrote:

What I find is missing most in the docs is a top-down description of the Query DSL: what is the possible structure of a search request body. It's all in the docs, but split into many little pieces which makes the "vertical" view (as Shay puts it) difficult to grasp. This is even complicated further by the fact that some pieces can go in multiple places (a query within a filter for example).

What would be very useful is a formal description of the Query DSL (similar to what an XSD offers to XML). No, not that complicated, but it would be complete (covers all possibilities) and would show what is required and what is optional for each "node" type. And, while I'm dreaming, it would also be clickable so you could drill-down.

The facet documentation has a formal description (uses and such) but is incomplete (each example only shows part of what is available): Elasticsearch Platform — Find real-time answers at scale | Elastic

I'm not too familiar with the code that parses the queries, but could such a document be generated from the code? Maybe using annotations or introspection or whatever, that would make it easily maintainable...

I'd like to contribute if that makes any sense to anyone.

Philippe

On Fri, May 20, 2011 at 16:21, Shay Banon shay.banon@elasticsearch.com wrote:

What we can do is start a section in the site with samples. I prefer not to have several places of docs around for elasticsearch. And the site is, effectively, a wiki (anybody can edit and send pull requests).
On Friday, May 20, 2011 at 11:19 PM, Karussell wrote:

I completely agree with that :slight_smile:

but for the sake of just working examples (sometimes user needs a
lot!) a wiki would be very nice - even if sprinkled here and there.

I started a wiki here:

http://elastic.wikispaces.com

29

Hi Clinton,

Thank you for your post. It is very useful. Can I user crawler.pl as a generic crawler for any url?? I'm not a perl guy, but the code seems to crawling based on

30

Hi Nehatha

Thank you for your post. It is very useful. Can I user crawler.pl as a
generic crawler for any url?? I'm not a perl guy, but the code seems to
crawling based on

That particular crawler uses xpath to express the content that should be
extracted from each page.

If that is true, Can you provide a generic crawler code plz...

I'm afraid that is left as an exercise for the reader :wink:

clint

31

Thank you for the reply Clinton.

Can you suggest best php/java crawler api's, best suited to feed ES.

Thanks,
Nehatha