OT: Formal description documentation

plaflamme · May 21, 2011, 2:37am

Hi,

I've forked the es.org site to start the formal description of the Query DSL
mentioned here[1].

I'm trying to make blocks similar to the normal code blocks but with the
ability to have links inside. The normal code blocks use a

, so it's a

no-go if I want links. I tried to make a  with appropriate style and

classes, but the Textile renderer is eating it, so I'm not sure what the

best approach would be here.
Can anyone with experience with the site/textile/jekyll etc. help me out?
My test documentation is here[2] (scroll all the way down).
Thanks,

Philippe

[1]

http://groups.google.com/a/elasticsearch.com/group/users/msg/efeecc7b64084672

[2]


  
      github.com
  
  
    plaflamme/elasticsearch.github.com/blob/master/guide/reference/api/search/facets/index.textile
---
layout: guide
title: Search API - Facets
cat: guide
sidebar: reference_api_search_facets
---

The usual purpose of a full-text search engine is to return a small number of documents matching your query.

_Facets_ provide aggregated data based on a search query. In the simple case, a facet can return _facet counts_ for various _facet values_ for a specific _field_. ElasticSearch supports more advanced facet implementations, such as "statistical":/guide/reference/api/search/facets/statistical-facet.html or "date histogram":/guide/reference/api/search/facets/date-histogram-facet.html facets.

The field used for facet calculations must be of type numeric, date/time or analyzed as a single token. You can give the facet an arbitrary _name_ and return multiple facets in one request.

For example, let's suppose you have a number of articles with a field called @tags@. Facet can return counts for the most popular tags across the documents matching your query — or across all documents in the index.

Let's check out this simple example. We will store some example data first:

<pre class="prettyprint lang-bash">
curl -X DELETE "http://localhost:9200/articles"
curl -X POST "http://localhost:9200/articles/article" -d '{"title" : "One",   "tags" : ["foo"]}'


  This file has been truncated. show original

kimchy · May 21, 2011, 2:08pm

Mayeb notextile will help? (http://textile.thresholdstate.com/).
On Saturday, May 21, 2011 at 5:37 AM, Philippe Laflamme wrote:

Hi,

I've forked the es.org site to start the formal description of the Query DSL mentioned here[1].

I'm trying to make blocks similar to the normal code blocks but with the ability to have links inside. The normal code blocks use a

, so it's a no-go if I want links. I tried to make a  with appropriate style and classes, but the Textile renderer is eating it, so I'm not sure what the best approach would be here.
Can anyone with experience with the site/textile/jekyll etc. help me out?
My test documentation is here[2] (scroll all the way down).
Thanks,

Philippe

[1] http://groups.google.com/a/elasticsearch.com/group/users/msg/efeecc7b64084672

[2] https://github.com/plaflamme/elasticsearch.github.com/blob/master/guide/reference/api/search/facets/index.textile

Nicolas_Lalevee · May 21, 2011, 4:19pm

Le 21 mai 2011 à 04:37, Philippe Laflamme a écrit :

Hi,

I've forked the es.org site to start the formal description of the Query DSL mentioned here[1].

I'm trying to make blocks similar to the normal code blocks but with the ability to have links inside. The normal code blocks use a
, so it's a no-go if I want links. I tried to make a  with appropriate style and classes, but the Textile renderer is eating it, so I'm not sure what the best approach would be here.
Can anyone with experience with the site/textile/jekyll etc. help me out?
My test documentation is here[2] (scroll all the way down).

If I may, if a formal documentation should exists, I think it should not be overloaded with the json syntax. I amy be probably better to write the abstract grammar.

And if I can suggest a presentation, I think the Ivy documentation about the format of an ivy.xml is quite friendly (since I'm an Ivy committer, I might be subjective here ).
See the tree of the left of the doc [1]. Each node of the tree of the table of content of the doc corresponds to an xml node in the ivy.xml file. Then on each page, some html tables describes each possible subnode, with links to the page of the subnode. This makes easy to know where a particular node should be placed in the final document, and easy to find the documentation of a node.

Nicolas

[1] Ivy Files | Apache Ivy™ Documentation

Clinton_Gormley · May 21, 2011, 6:01pm

Hiya

I've forked the es.org site to start the formal description of the
Query DSL mentioned here[1].

I used to maintain a semi-formal query dsl spec, but it is way out of
date now:

http://elasticsearch-users.115913.n3.nabble.com/Formal-spec-for-Query-DSL-tp217565p217565.html

clint

plaflamme · May 21, 2011, 6:30pm

If I may, if a formal documentation should exists, I think it should not be
overloaded with the json syntax. I amy be probably better to write the
abstract grammar.

Yeah, I agree that not using any JSON would be better. But it's also
important that the result is clear and easy to transfer back to an actual
JSON syntax. We'll have to find a balance between both: not using any JSON
and easy transition from doc to JSON.

I thought of documenting the grammar first, but then I decided that working
with an actual use-case would make it easier to come up with the grammar.
Have any idea where you'd like to start?

And if I can suggest a presentation, I think the Ivy documentation about
the format of an ivy.xml is quite friendly (since I'm an Ivy committer, I
might be subjective here ).
See the tree of the left of the doc [1]. Each node of the tree of the table
of content of the doc corresponds to an xml node in the ivy.xml file. Then
on each page, some html tables describes each possible subnode, with links
to the page of the subnode. This makes easy to know where a particular node
should be placed in the final document, and easy to find the documentation
of a node.

That's pretty much what I had in mind. Making a tree structure for ES is
somewhat tricky since some node types can appear in multiple
places. Otherwise, each node would have its own page (which is pretty much
already the case) and each page would have the same structure. Something
along the lines of:

Description
Textual description of what the node does and why you'd use it
Formal Description
Pretty much what your Ivy example shows. A formal description of each node
that states what is required, optional, etc. A table structure is great.
Nodes must be linked together.
Examples
Lots of 'em.
Notes
I noticed that some of nodes have notes about caching and performance.
This is where this stuff could be captured

Is the Ivy documentation generated from an XSD? Since it's XML and probably
already stated in the XSD, I would guess it be the case. I especially don't
think we should be writing an XSD though!

Thanks,
Philippe

plaflamme · May 21, 2011, 6:32pm

That's great! This is exactly what I had in mind. We'll definitely want to
use this for starting a formal description.

Thanks,
Philippe

On Sat, May 21, 2011 at 14:01, Clinton Gormley clinton@iannounce.co.ukwrote:

Hiya

I've forked the es.org site to start the formal description of the
Query DSL mentioned here[1].

I used to maintain a semi-formal query dsl spec, but it is way out of
date now:

http://elasticsearch-users.115913.n3.nabble.com/Formal-spec-for-Query-DSL-tp217565p217565.html

clint

plaflamme · May 21, 2011, 6:41pm

Nope. Now I tried applying style to the

like so:

p{white-space:pre}. content

And the result is a simple

without any style attribute. Could this be a
github-specific issue? Would jekyll render this differently maybe?

Philippe

On Sat, May 21, 2011 at 10:08, Shay Banon shay.banon@elasticsearch.comwrote:

Mayeb notextile will help? (http://textile.thresholdstate.com/).

On Saturday, May 21, 2011 at 5:37 AM, Philippe Laflamme wrote:

Hi,

I've forked the es.org site to start the formal description of the Query
DSL mentioned here[1].

I'm trying to make blocks similar to the normal code blocks but with the
ability to have links inside. The normal code blocks use a
, so it's a

no-go if I want links. I tried to make a  with appropriate style and

classes, but the Textile renderer is eating it, so I'm not sure what the

best approach would be here.
Can anyone with experience with the site/textile/jekyll etc. help me out?
My test documentation is here[2] (scroll all the way down).
Thanks,

Philippe

[1]

http://groups.google.com/a/elasticsearch.com/group/users/msg/efeecc7b64084672

[2]

https://github.com/plaflamme/elasticsearch.github.com/blob/master/guide/reference/api/search/facets/index.textile

Clinton_Gormley · May 21, 2011, 6:45pm

I thought of documenting the grammar first, but then I decided that
working with an actual use-case would make it easier to come up with
the grammar. Have any idea where you'd like to start?

I'm working on an alternate more compact syntax which I plan to
implement in the Perl API - hopefully it will gain wider acceptance.

While doing this, I've noticed that the various filter and query types
can be subdivided into various logical groups, eg:

full text queries (text*, field, query_string)
term based queries (term, terms, prefix, wildcard, fuzzy, range,
span*)
meta-queries (fuzzy_like_this, more_like_this, top|has_children)
logical combinators (bool, and|or|not, dis_max, boosting)
custom: custom_score
etc

My intention is to write some "navigation" docs to explain the
difference between term and text clauses, queries and filters, and which
query/filter to use for which purpose.

Of course, if somebody gets there before me...

clint

Nicolas_Lalevee · May 21, 2011, 6:59pm

Le 21 mai 2011 à 20:30, Philippe Laflamme a écrit :

Yeah, I agree that not using any JSON would be better. But it's also important that the result is clear and easy to transfer back to an actual JSON syntax. We'll have to find a balance between both: not using any JSON and easy transition from doc to JSON.

To my mind, if there are examples, the formal description doesn't need to precisely describe the JSON syntax. My user experience of the Ivy documentation (and also Ant's one) is about reading the formal spec to understand what it is about, and then for the syntax I always go see the examples. This may be just me.

That's pretty much what I had in mind. Making a tree structure for ES is somewhat tricky since some node types can appear in multiple places. Otherwise, each node would have its own page (which is pretty much already the case) and each page would have the same structure. Something along the lines of:

Description
Textual description of what the node does and why you'd use it

Formal Description
Pretty much what your Ivy example shows. A formal description of each node that states what is required, optional, etc. A table structure is great. Nodes must be linked together.

Examples
Lots of 'em.

Notes
I noticed that some of nodes have notes about caching and performance. This is where this stuff could be captured

Seems nice !

Is the Ivy documentation generated from an XSD? Since it's XML and probably already stated in the XSD, I would guess it be the case. I especially don't think we should be writing an XSD though!

It's not generated, no. I know xsddoc [1] which can manage xsd related documentation. But the result is not that sexy and easy to understand. And I cannot imagine maintaing the entire doc of Ivy in some xsd:comment of Ivy's xsd. Currently the doc is not hard to maintain. I have to admit that sometimes some piece of the doc is forgotten to be updated, there's still probably some holes. I don't think this is that critical, and I have seen people easily provide patch for the doc.

Nicolas

[1] xsddoc - xsddoc

plaflamme · May 23, 2011, 2:44am

Hi,

Made some improvements to the formal descriptions after some very helpful
comments.

I've reformatted using a table structure, much cleaner (thanks Nicolas for
that suggestion). The "facets" node comes out kinda weird because of its
dynamic nature[1]. I did the "terms" facet node which comes out pretty
nicely[2], I think.

Would something like this be a welcomed addition to the documentation? I'm
not sure about the structure : whether this should go at the top, stay at
the bottom or go somewhere else completely. Any comments/suggestions
welcomed.

Thanks,
Philippe
[1]

github.com

plaflamme/elasticsearch.github.com/blob/master/guide/reference/api/search/facets/index.textile

---
layout: guide
title: Search API - Facets
cat: guide
sidebar: reference_api_search_facets
---

The usual purpose of a full-text search engine is to return a small number of documents matching your query.

_Facets_ provide aggregated data based on a search query. In the simple case, a facet can return _facet counts_ for various _facet values_ for a specific _field_. ElasticSearch supports more advanced facet implementations, such as "statistical":/guide/reference/api/search/facets/statistical-facet.html or "date histogram":/guide/reference/api/search/facets/date-histogram-facet.html facets.

The field used for facet calculations must be of type numeric, date/time or analyzed as a single token. You can give the facet an arbitrary _name_ and return multiple facets in one request.

For example, let's suppose you have a number of articles with a field called @tags@. Facet can return counts for the most popular tags across the documents matching your query — or across all documents in the index.

Let's check out this simple example. We will store some example data first:

<pre class="prettyprint lang-bash">
curl -X DELETE "http://localhost:9200/articles"
curl -X POST "http://localhost:9200/articles/article" -d '{"title" : "One",   "tags" : ["foo"]}'

This file has been truncated. show original

[2]

github.com

plaflamme/elasticsearch.github.com/blob/master/guide/reference/api/search/facets/terms-facet.textile

---
layout: guide
title: Search API - Terms Facet
cat: guide
sidebar: reference_api_search_facets
---

p. Allow to specify field facets that return the N most frequent terms. For example:

<pre class="prettyprint lang-js">
{
    "query" : {
        "match_all" : {  }
    },
    "facets" : {
        "tag" : {
            "terms" : {
                "field" : "tag",
                "size" : 10
            }

This file has been truncated. show original

On Fri, May 20, 2011 at 22:37, Philippe Laflamme <
philippe.laflamme@obiba.org> wrote:

Hi,

I've forked the es.org site to start the formal description of the Query
DSL mentioned here[1].

I'm trying to make blocks similar to the normal code blocks but with the
ability to have links inside. The normal code blocks use a

, so it's a

no-go if I want links. I tried to make a  with appropriate style and

classes, but the Textile renderer is eating it, so I'm not sure what the

best approach would be here.
Can anyone with experience with the site/textile/jekyll etc. help me out?
My test documentation is here[2] (scroll all the way down).
Thanks,

Philippe

[1]

http://groups.google.com/a/elasticsearch.com/group/users/msg/efeecc7b64084672

[2]

https://github.com/plaflamme/elasticsearch.github.com/blob/master/guide/reference/api/search/facets/index.textile

Topic		Replies	Views
Terms-stats facet and aggregation Elasticsearch	1	368	July 6, 2017
Faceted Navigation With Labels Elasticsearch	1	336	July 6, 2017
Opinion needed about Document design for recursive sub-objects/documents Elasticsearch	4	1562	July 6, 2017
Facets have got me down Elasticsearch	11	584	July 6, 2017
Help with grouping facet Elasticsearch	1	265	July 6, 2017

OT: Formal description documentation

Related topics