I used ES for one year I like all features it provides except the query
DSL. The json style DSL is not intuitive and was confused me for a while
when I was putting multiple conditions together at very beginning.
enlightened by Jira, I am thinking why don't ES create a DSL like:
project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
created DESC, cf[10514] DESC
this kind of external DSL needs a parser which might be the reason ES chose
json style internal DSL at beginning. But from user perspective, the
external DSL is much better to understand and use. I know there are SDKs
for most of popular languages which relieves people's life a lot, but an
easier query DSL is still valuable to people who want to directly play with
ES's rest api.
you can write your own plugin, which simply registers a custom parser on
start up of elasticsearch, which is able to parse the above statement. So,
basically there is nothing stopping you from doing this. You could embed
this parser in the JSON DSL, but if you do not like it, you could simply
write your own REST HTTP action, which parses the above as text without
JSON being involved.
Take a look at some rest actions and at the QueryParser interface and its
implementions.
Elasticsearch tries to be as modular as possible in order to give you
exactly this flexibility, that you can write your own custom
implementations, in case you are unhappy with the status quo. And hopefully
open source them
I used ES for one year I like all features it provides except the query
DSL. The json style DSL is not intuitive and was confused me for a while
when I was putting multiple conditions together at very beginning.
enlightened by Jira, I am thinking why don't ES create a DSL like:
project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
created DESC, cf[10514] DESC
this kind of external DSL needs a parser which might be the reason ES
chose json style internal DSL at beginning. But from user perspective, the
external DSL is much better to understand and use. I know there are SDKs
for most of popular languages which relieves people's life a lot, but an
easier query DSL is still valuable to people who want to directly play with
ES's rest api.
In the DSL you give, there are some subtle issues which makes it hard for
implementing a parser and correct query execution.
For example, the word "New feature": is it just a phrase, or is it two
terms? Is a span query for the two terms also valid?
Another issue is well-formed input data typing. Is "ES" in the query an
input of type "string"? And is "3.1" is double value or also a string type?
Can types always be inferenced? What about date parsing, will date types be
assigned automatically? Maybe by looking up the ES mapping (well, ES
parsers do that already) ? Or, how to decide if "true" is the string "true"
or the boolean constant? ES catches this but I mean the general case of an
external DSL is dealing with JSON data type flaws. You are forced to
re-implement the ES parser for all these nasty pitfalls.
How to declare facets and filters in an external DSL? Or multi phrase and
multi wildcard searching? It's not straightforward if there is simply no
context information how to execute such things. How do you parse and
translate wildcards mixed/nested with phrases such as "scien* 'week*
magazin*'"? I was surprised how many folks are trained to use wildcards
excessively. It is only possible to replace such "bad queries" with
heuristics that can be executed on ES with high performance, with ranked
results etc.
My experience so far is, at least for CQL, because it is a weak typed query
language (it does not have a notion of input data types) , that external
query languages must really be able to match the power of Lucene/ES
features, or you get into trouble implementing simplifications, fallbacks,
and shortcuts all the way.
So my favorite is still ES DSL, and for establishing simple searches, there
are special featured query types that are designed for simplified free form
input. E.g. ES DSL query type "query_string" understands the Lucene syntax,
or the "match" query, and now we also have the "simple_query"
I'm interested in OpenSearch for ES http://www.opensearch.org/ so if anyone
is working on this, it would be nice to know.
Thanks Alex and Jorg. Frankly speaking I didn't think of details of
implementing such external DSL in ES. I am just from a user's perspective
thinking of what's the best user experience.
Jorg raises a good question that how to distinguish data type given the
nature of schemaless in ES, the DSL may have to have some keywords(e.g int,
long, date) to tell data type explicitly if it's important, otherwise all
data are treated as string. For other questions I can not answer because
some of features like facet I have not even used. I know it's not
straightforward to do such thing, but I do think if JSON DSL can solve
these problems then the external DSL can also do because it's more flexible
like a programming language. I have been playing with some NoSql databases
as well as search engine like ES, an impression I got is every software has
its own search DSL which are not user friendly. Though I think it might be
impossible for this area to agree on a standard similar to SQL given these
software focus on different parts and usually vary a lot, I think using a
SQL style DSL without join and sub-select can make user's life much easier.
On Friday, December 20, 2013 4:32:30 AM UTC-8, Jörg Prante wrote:
Just some food for thought:
In the DSL you give, there are some subtle issues which makes it hard for
implementing a parser and correct query execution.
For example, the word "New feature": is it just a phrase, or is it two
terms? Is a span query for the two terms also valid?
Another issue is well-formed input data typing. Is "ES" in the query an
input of type "string"? And is "3.1" is double value or also a string type?
Can types always be inferenced? What about date parsing, will date types be
assigned automatically? Maybe by looking up the ES mapping (well, ES
parsers do that already) ? Or, how to decide if "true" is the string "true"
or the boolean constant? ES catches this but I mean the general case of an
external DSL is dealing with JSON data type flaws. You are forced to
re-implement the ES parser for all these nasty pitfalls.
How to declare facets and filters in an external DSL? Or multi phrase and
multi wildcard searching? It's not straightforward if there is simply no
context information how to execute such things. How do you parse and
translate wildcards mixed/nested with phrases such as "scien* 'week*
magazin*'"? I was surprised how many folks are trained to use wildcards
excessively. It is only possible to replace such "bad queries" with
heuristics that can be executed on ES with high performance, with ranked
results etc.
My experience so far is, at least for CQL, because it is a weak typed
query language (it does not have a notion of input data types) , that
external query languages must really be able to match the power of
Lucene/ES features, or you get into trouble implementing simplifications,
fallbacks, and shortcuts all the way.
In the DSL you give, there are some subtle issues which makes it hard for
implementing a parser and correct query execution.
For example, the word "New feature": is it just a phrase, or is it two
terms? Is a span query for the two terms also valid?
Another issue is well-formed input data typing. Is "ES" in the query an
input of type "string"? And is "3.1" is double value or also a string type?
Can types always be inferenced? What about date parsing, will date types be
assigned automatically? Maybe by looking up the ES mapping (well, ES
parsers do that already) ? Or, how to decide if "true" is the string "true"
or the boolean constant? ES catches this but I mean the general case of an
external DSL is dealing with JSON data type flaws. You are forced to
re-implement the ES parser for all these nasty pitfalls.
How to declare facets and filters in an external DSL? Or multi phrase and
multi wildcard searching? It's not straightforward if there is simply no
context information how to execute such things. How do you parse and
translate wildcards mixed/nested with phrases such as "scien* 'week*
magazin*'"? I was surprised how many folks are trained to use wildcards
excessively. It is only possible to replace such "bad queries" with
heuristics that can be executed on ES with high performance, with ranked
results etc.
My experience so far is, at least for CQL, because it is a weak typed
query language (it does not have a notion of input data types) , that
external query languages must really be able to match the power of
Lucene/ES features, or you get into trouble implementing simplifications,
fallbacks, and shortcuts all the way.
That is a special endpoint search that understands a simplified query
parameter. The query parameter can be anything, for example an SQL-like
syntax like xin zhang suggested.
The effect would be the integration of ES into 3rd party products that
offer simplified search.
Opensearch is known from browser search forms like Firefox search plugins.
This would of course not replace ES official clients, but some PoCs were
easier ("let's get some ES results in our product and show it in the
browser").
All in all, what the plugin would have to do is a bit of processing
simplified queries and reformatting ES JSON result, preferably as Atom
feeds, both XML and JSON. So a reverse HTTP proxy could just pass
parameters and results.
I had same exact requirement of doing a user facing query dsl on top of elasticsearch dsl. I had a look at SearchRetrieve and it looks quite promising, I wish it was open sourced!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
