# Did ES ever consider replacing current query DSL with an external DSL?
**URL:** https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956
**Category:** Elasticsearch
**Created:** [December 19, 2013, 7:05pm UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956 "2013-12-19T19:05:17Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@xin\_zhang](https://discuss.elastic.co/u/xin_zhang)
#### Post date: [December 19, 2013, 7:05pm UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/1 "2013-12-19T19:05:17Z")
I used ES for one year I like all features it provides except the query
DSL. The json style DSL is not intuitive and was confused me for a while
when I was putting multiple conditions together at very beginning.
enlightened by Jira, I am thinking why don't ES create a DSL like:
project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
created DESC, cf[10514] DESC
this kind of external DSL needs a parser which might be the reason ES chose
json style internal DSL at beginning. But from user perspective, the
external DSL is much better to understand and use. I know there are SDKs
for most of popular languages which relieves people's life a lot, but an
easier query DSL is still valuable to people who want to directly play with
ES's rest api.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7650a534-fab1-4ccf-addd-a59085f35f08%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7650a534-fab1-4ccf-addd-a59085f35f08%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@spinscale](https://discuss.elastic.co/u/spinscale)
#### Post date: [December 20, 2013, 10:55am UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/2 "2013-12-20T10:55:09Z")
Hey,
you can write your own plugin, which simply registers a custom parser on
start up of elasticsearch, which is able to parse the above statement. So,
basically there is nothing stopping you from doing this. You could embed
this parser in the JSON DSL, but if you do not like it, you could simply
write your own REST HTTP action, which parses the above as text without
JSON being involved.
Take a look at some rest actions and at the QueryParser interface and its
implementions.
Elasticsearch tries to be as modular as possible in order to give you
exactly this flexibility, that you can write your own custom
implementations, in case you are unhappy with the status quo. And hopefully
open source them 馃檪
--Alex
On Thu, Dec 19, 2013 at 8:05 PM, xin zhang [xing5820@gmail.com](mailto:xing5820@gmail.com) wrote:
> I used ES for one year I like all features it provides except the query
> DSL. The json style DSL is not intuitive and was confused me for a while
> when I was putting multiple conditions together at very beginning.
>
> enlightened by Jira, I am thinking why don't ES create a DSL like:
>
> project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
> created DESC, cf[10514] DESC
>
> this kind of external DSL needs a parser which might be the reason ES
> chose json style internal DSL at beginning. But from user perspective, the
> external DSL is much better to understand and use. I know there are SDKs
> for most of popular languages which relieves people's life a lot, but an
> easier query DSL is still valuable to people who want to directly play with
> ES's rest api.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/7650a534-fab1-4ccf-addd-a59085f35f08%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7650a534-fab1-4ccf-addd-a59085f35f08%40googlegroups.com)
> .
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM\_ofHn8ApoELzeqSJG2Zrfu2bTJp2nBeDXpP7AP%3DSLAMA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_ofHn8ApoELzeqSJG2Zrfu2bTJp2nBeDXpP7AP%3DSLAMA%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [December 20, 2013, 12:32pm UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/3 "2013-12-20T12:32:30Z")
Just some food for thought:
In the DSL you give, there are some subtle issues which makes it hard for
implementing a parser and correct query execution.
For example, the word "New feature": is it just a phrase, or is it two
terms? Is a span query for the two terms also valid?
Another issue is well-formed input data typing. Is "ES" in the query an
input of type "string"? And is "3.1" is double value or also a string type?
Can types always be inferenced? What about date parsing, will date types be
assigned automatically? Maybe by looking up the ES mapping (well, ES
parsers do that already) ? Or, how to decide if "true" is the string "true"
or the boolean constant? ES catches this but I mean the general case of an
external DSL is dealing with JSON data type flaws. You are forced to
re-implement the ES parser for all these nasty pitfalls.
How to declare facets and filters in an external DSL? Or multi phrase and
multi wildcard searching? It's not straightforward if there is simply no
context information how to execute such things. How do you parse and
translate wildcards mixed/nested with phrases such as "scien\* 'week\*
magazin\*'"? I was surprised how many folks are trained to use wildcards
excessively. It is only possible to replace such "bad queries" with
heuristics that can be executed on ES with high performance, with ranked
results etc.
I have written a CQL parser
[http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.pdfto](http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.pdfto)
generate ES DSL, but for Java API only. If anyone is interested in
adding a CQL parser as a REST action, I could offer it as a plugin. It is
of course not perfect, I'm not very satisfied with the result.
My experience so far is, at least for CQL, because it is a weak typed query
language (it does not have a notion of input data types) , that external
query languages must really be able to match the power of Lucene/ES
features, or you get into trouble implementing simplifications, fallbacks,
and shortcuts all the way.
So my favorite is still ES DSL, and for establishing simple searches, there
are special featured query types that are designed for simplified free form
input. E.g. ES DSL query type "query\_string" understands the Lucene syntax,
or the "match" query, and now we also have the "simple\_query"
>
>
> This adds support for Lucene's SimpleQueryParser by adding a new type
> of query c鈥lled the \`simple\_query\_string\`. The \`simple\_query\_string\`
> query is designed to be able to parse human-entered queries without
> throwing any exceptions.
>
> Resolves #4159.
I'm interested in OpenSearch for ES [http://www.opensearch.org/](http://www.opensearch.org/) so if anyone
is working on this, it would be nice to know.
J枚rg
On Thu, Dec 19, 2013 at 8:05 PM, xin zhang [xing5820@gmail.com](mailto:xing5820@gmail.com) wrote:
> project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
> created DESC, cf[10514] DESC
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGhkOE038spXw1Tmm0zLv%3DS%3D8YAztaMZEYK-hVNwhMB5Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGhkOE038spXw1Tmm0zLv%3DS%3D8YAztaMZEYK-hVNwhMB5Q%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@xin\_zhang](https://discuss.elastic.co/u/xin_zhang)
#### Post date: [December 20, 2013, 11:04pm UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/4 "2013-12-20T23:04:10Z")
Thanks Alex and Jorg. Frankly speaking I didn't think of details of
implementing such external DSL in ES. I am just from a user's perspective
thinking of what's the best user experience.
Jorg raises a good question that how to distinguish data type given the
nature of schemaless in ES, the DSL may have to have some keywords(e.g int,
long, date) to tell data type explicitly if it's important, otherwise all
data are treated as string. For other questions I can not answer because
some of features like facet I have not even used. I know it's not
straightforward to do such thing, but I do think if JSON DSL can solve
these problems then the external DSL can also do because it's more flexible
like a programming language. I have been playing with some NoSql databases
as well as search engine like ES, an impression I got is every software has
its own search DSL which are not user friendly. Though I think it might be
impossible for this area to agree on a standard similar to SQL given these
software focus on different parts and usually vary a lot, I think using a
SQL style DSL without join and sub-select can make user's life much easier.
On Friday, December 20, 2013 4:32:30 AM UTC-8, J枚rg Prante wrote:
> Just some food for thought:
>
> In the DSL you give, there are some subtle issues which makes it hard for
> implementing a parser and correct query execution.
>
> For example, the word "New feature": is it just a phrase, or is it two
> terms? Is a span query for the two terms also valid?
>
> Another issue is well-formed input data typing. Is "ES" in the query an
> input of type "string"? And is "3.1" is double value or also a string type?
> Can types always be inferenced? What about date parsing, will date types be
> assigned automatically? Maybe by looking up the ES mapping (well, ES
> parsers do that already) ? Or, how to decide if "true" is the string "true"
> or the boolean constant? ES catches this but I mean the general case of an
> external DSL is dealing with JSON data type flaws. You are forced to
> re-implement the ES parser for all these nasty pitfalls.
>
> How to declare facets and filters in an external DSL? Or multi phrase and
> multi wildcard searching? It's not straightforward if there is simply no
> context information how to execute such things. How do you parse and
> translate wildcards mixed/nested with phrases such as "scien\* 'week\*
> magazin\*'"? I was surprised how many folks are trained to use wildcards
> excessively. It is only possible to replace such "bad queries" with
> heuristics that can be executed on ES with high performance, with ranked
> results etc.
>
> I have written a CQL parser
> [http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.pdfto](http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.pdfto) generate ES DSL, but for Java API only. If anyone is interested in
> adding a CQL parser as a REST action, I could offer it as a plugin. It is
> of course not perfect, I'm not very satisfied with the result.
>
> My experience so far is, at least for CQL, because it is a weak typed
> query language (it does not have a notion of input data types) , that
> external query languages must really be able to match the power of
> Lucene/ES features, or you get into trouble implementing simplifications,
> fallbacks, and shortcuts all the way.
>
> So my favorite is still ES DSL, and for establishing simple searches,
> there are special featured query types that are designed for simplified
> free form input. E.g. ES DSL query type "query\_string" understands the
> Lucene syntax, or the "match" query, and now we also have the
> "simple\_query" [Add new `simple_query_string` query type by dakrone 路 Pull Request #4402 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/pull/4402)
>
> I'm interested in OpenSearch for ES [http://www.opensearch.org/](http://www.opensearch.org/) so if
> anyone is working on this, it would be nice to know.
>
> J枚rg
>
> On Thu, Dec 19, 2013 at 8:05 PM, xin zhang \<[xing...@gmail.com](mailto:xing...@gmail.com)\
>
> > wrote:
> >
> > project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
> > created DESC, cf[10514] DESC
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0a5a07e4-666b-4498-a781-c48882fae2c6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0a5a07e4-666b-4498-a781-c48882fae2c6%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [December 21, 2013, 9:06am UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/5 "2013-12-21T09:06:47Z")
J枚rg,
is opensearch active? The last time I checked it it seemed pretty much dead.
Luk谩拧
Dne 20.12.2013 13:32 "[joergprante@gmail.com](mailto:joergprante@gmail.com)" [joergprante@gmail.com](mailto:joergprante@gmail.com)
napsal(a):
> Just some food for thought:
>
> In the DSL you give, there are some subtle issues which makes it hard for
> implementing a parser and correct query execution.
>
> For example, the word "New feature": is it just a phrase, or is it two
> terms? Is a span query for the two terms also valid?
>
> Another issue is well-formed input data typing. Is "ES" in the query an
> input of type "string"? And is "3.1" is double value or also a string type?
> Can types always be inferenced? What about date parsing, will date types be
> assigned automatically? Maybe by looking up the ES mapping (well, ES
> parsers do that already) ? Or, how to decide if "true" is the string "true"
> or the boolean constant? ES catches this but I mean the general case of an
> external DSL is dealing with JSON data type flaws. You are forced to
> re-implement the ES parser for all these nasty pitfalls.
>
> How to declare facets and filters in an external DSL? Or multi phrase and
> multi wildcard searching? It's not straightforward if there is simply no
> context information how to execute such things. How do you parse and
> translate wildcards mixed/nested with phrases such as "scien\* 'week\*
> magazin\*'"? I was surprised how many folks are trained to use wildcards
> excessively. It is only possible to replace such "bad queries" with
> heuristics that can be executed on ES with high performance, with ranked
> results etc.
>
> I have written a CQL parser
> [http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.pdfto](http://docs.oasis-open.org/search-ws/searchRetrieve/v1.0/os/part5-cql/searchRetrieve-v1.0-os-part5-cql.pdfto) generate ES DSL, but for Java API only. If anyone is interested in
> adding a CQL parser as a REST action, I could offer it as a plugin. It is
> of course not perfect, I'm not very satisfied with the result.
>
> My experience so far is, at least for CQL, because it is a weak typed
> query language (it does not have a notion of input data types) , that
> external query languages must really be able to match the power of
> Lucene/ES features, or you get into trouble implementing simplifications,
> fallbacks, and shortcuts all the way.
>
> So my favorite is still ES DSL, and for establishing simple searches,
> there are special featured query types that are designed for simplified
> free form input. E.g. ES DSL query type "query\_string" understands the
> Lucene syntax, or the "match" query, and now we also have the
> "simple\_query" [Add new `simple_query_string` query type by dakrone 路 Pull Request #4402 路 elastic/elasticsearch 路 GitHub](https://github.com/elasticsearch/elasticsearch/pull/4402)
>
> I'm interested in OpenSearch for ES [http://www.opensearch.org/](http://www.opensearch.org/) so if
> anyone is working on this, it would be nice to know.
>
> J枚rg
>
> On Thu, Dec 19, 2013 at 8:05 PM, xin zhang [xing5820@gmail.com](mailto:xing5820@gmail.com) wrote:
>
> > project = ES AND issuetype = "New Feature" AND fixVersion = 3.1 ORDER BY
> > created DESC, cf[10514] DESC
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGhkOE038spXw1Tmm0zLv%3DS%3D8YAztaMZEYK-hVNwhMB5Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGhkOE038spXw1Tmm0zLv%3DS%3D8YAztaMZEYK-hVNwhMB5Q%40mail.gmail.com)
> > .
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbqrVMq%3DoM5MS%3DwW2tJqFxC8FwC-d7R9CG4Hm1ihjfgtQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbqrVMq%3DoM5MS%3DwW2tJqFxC8FwC-d7R9CG4Hm1ihjfgtQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [December 21, 2013, 2:28pm UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/6 "2013-12-21T14:28:03Z")
Opensearch is a bit silent, but not defunct, there is a low activity
mailing list
[https://groups.google.com/forum/#!forum/opensearch](https://groups.google.com/forum/#!forum/opensearch)
My impression is there is not much more Opensearch can be developed into,
as a "de facto" very lightweight standard.
Example
[http://chroniclingamerica.loc.gov/search/pages/results/?proxtext=ducks&format=json](http://chroniclingamerica.loc.gov/search/pages/results/?proxtext=ducks&format=json)
That is a special endpoint search that understands a simplified query
parameter. The query parameter can be anything, for example an SQL-like
syntax like xin zhang suggested.
The effect would be the integration of ES into 3rd party products that
offer simplified search.
Opensearch is known from browser search forms like Firefox search plugins.
This would of course not replace ES official clients, but some PoCs were
easier ("let's get some ES results in our product and show it in the
browser").
OpenSearch integration into Wikis, Blogs
[http://www.opensearch.org/Community/OpenSearch\_software](http://www.opensearch.org/Community/OpenSearch_software)
OpenSearch Geo
[http://www.weichand.de/2010/10/27/opensearch-geo-die-einfache-raeumliche-suche/](http://www.weichand.de/2010/10/27/opensearch-geo-die-einfache-raeumliche-suche/)
All in all, what the plugin would have to do is a bit of processing
simplified queries and reformatting ES JSON result, preferably as Atom
feeds, both XML and JSON. So a reverse HTTP proxy could just pass
parameters and results.
J枚rg
On Sat, Dec 21, 2013 at 10:06 AM, Luk谩拧 Vl膷ek [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:
> J枚rg,
> is opensearch active? The last time I checked it it seemed pretty much
> dead.
> Luk谩拧
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFM1HQ5\_5PtGRrSazBzGns-ZEX-rwtPqrNXUL2RgYJngw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFM1HQ5_5PtGRrSazBzGns-ZEX-rwtPqrNXUL2RgYJngw%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@techno\_shaft](https://discuss.elastic.co/u/techno_shaft)
#### Post date: [November 11, 2016, 5:51am UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/7 "2016-11-11T05:51:37Z")
I had same exact requirement of doing a user facing query dsl on top of elasticsearch dsl. I had a look at SearchRetrieve and it looks quite promising, I wish it was open sourced!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:05pm UTC](https://discuss.elastic.co/t/did-es-ever-consider-replacing-current-query-dsl-with-an-external-dsl/14956/8 "2017-07-05T22:05:55Z")