Top Hits aggregator from example not working for me


(Daniel Yim) #1

Hello, I am trying to follow the field collapse example on this page
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example,
which uses the new 1.3.0 top_hits aggregator to return the top scoring
document for a given criteria. To my knowledge, I set up the settings and
mappings correctly, but I am getting a strange error when I try to run the
query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"idx_analyzer": {
"tokenizer": "whitespace",
"filter": [
"lowercase",
"snowball",
"XYZSynFilter"
]
},
"sch_analyzer": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop"
]
},
"sch_comma_analyzer": {
"tokenizer": "CommaTokenizer",
"filter": [
"standard",
"lowercase",
"stop"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids virus, aids, retrovirology, hiv"
],
"expand": true,
"ignore_case": true
}
},
"tokenizer": {
"CommaTokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
},
"mappings": {
"employees": {
"properties": {
"fullName": {
"type": "string",
"search_analyzer": "sch_analyzer"
},
"specialty": {
"type": "string",
"search_analyzer": "sch_comma_analyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'
{
"fullName": "Don White",
"specialty": "Adult Retrovirology, aids, hiv"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'
{
"fullName": "Don White",
"specialty": "general practitioner, physician, general, primary care"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'
{
"fullName": "Don White",
"specialty": "icu, er"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'
{
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'
{
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'
{
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'
{
"fullName": "Carter Taylor",
"specialty": "neurosurgery, brain surgery, brain tumor"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'
{
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}'

Executing this search (per the example) gives me an error
curl -XGET
"http://localhost:9200/personsearch/employees/_search?pretty=true" -d'
{
"query": {
"query_string": {
"query": "icu"
}
},
"aggs": {
"most-rel-profile": {
"terms": {
"field": "profileName",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit": {
"max": {
"script": "_doc.score"
}
}
}
}
}
}'

{
"error": "SearchPhaseExecutionException[Failed to execute phase [query],
all shards failed; shardFailures
{[4gtV_OJESWWSgMnbTruKyA][personsearch][0]:
SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]:
SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]:
SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]:
SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]:
SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested:* ScriptException[dynamic scripting for
[mvel] disabled];* }]",
"status": 400
}

Does the dynamic scripting for [mvel] disabled have something to do with
it?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Rafał Kuć) #2

Hello!

The error is about scripting and dynamic scripting being disabled for MVEL. The simplest way to make it work is turning on dynamic scripting by adding script.disable_dynamic: false to your elasticsearch.yml file. However for production it is not recommended for security reasons. You can also try using scripting language that is sandboxed and allows for dynamic scripting, like groovy.

--

Regards,

Rafał Kuć

Performance Monitoring * Log Analytics * Search Analytics

Solr & Elasticsearch Support * http://sematext.com/

Hello, I am trying to follow the field collapse example on this page, which uses the new 1.3.0 top_hits aggregator to return the top scoring document for a given criteria. To my knowledge, I set up the settings and mappings correctly, but I am getting a strange error when I try to run the query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"

curl -XPUT "http://localhost:9200/personsearch" -d'

{

"settings": {

"index": {


  "analysis": {


    "analyzer": {


      "idx_analyzer": {


        "tokenizer": "whitespace",


        "filter": [


          "lowercase",


          "snowball",


          "XYZSynFilter"


        ]


      },


      "sch_analyzer": {


        "tokenizer": "standard",


        "filter": [


          "standard",


          "lowercase",


          "stop"


        ]


      },


      "sch_comma_analyzer": {


        "tokenizer": "CommaTokenizer",


        "filter": [


          "standard",


          "lowercase",


          "stop"


        ]


      }


    },


    "filter": {


      "XYZSynFilter": {


        "type": "synonym",


        "synonyms": [


          "aids virus, aids, retrovirology, hiv"


        ],


        "expand": true,


        "ignore_case": true


      }


    },


    "tokenizer": {


      "CommaTokenizer": {


        "type": "pattern",


        "pattern": ","


      }


    }


  }


}

},

"mappings": {

"employees": {


  "properties": {


    "fullName": {


      "type": "string",


      "search_analyzer": "sch_analyzer"


    },


    "specialty": {


      "type": "string",


      "search_analyzer": "sch_comma_analyzer"


    }


  }


}

}

}'

curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'

{

"fullName": "Don White",

"specialty": "Adult Retrovirology, aids, hiv"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'

{

"fullName": "Don White",

"specialty": "general practitioner, physician, general, primary care"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'

{

"fullName": "Don White",

"specialty": "icu, er"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'

{

"fullName": "Terrance Gartner",

"specialty": "oncology, cancer, research, tumor, polyp"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'

{

"fullName": "Terrance Gartner",

"specialty": "physician, general, GP, primary care, aids"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'

{

"fullName": "Terrance Gartner",

"specialty": "emergency care, icu, ambulance, er, urgent"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'

{

"fullName": "Carter Taylor",

"specialty": "neurosurgery, brain surgery, brain tumor"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'

{

"fullName": "Carter Taylor",

"specialty": "trauma, icu, emergency care, ER, urgent care"

}'

Executing this search (per the example) gives me an error

curl -XGET "http://localhost:9200/personsearch/employees/_search?pretty=true" -d'

{

"query": {

"query_string": {


  "query": "icu"


}

},

"aggs": {

"most-rel-profile": {


  "terms": {


    "field": "profileName",


    "order": {


      "top_hit": "desc"


    }


  },


  "aggs": {


    "top_tags_hits": {


      "top_hits": {}


    },


    "top_hit": {


      "max": {


        "script": "_doc.score"


      }


    }


  }


}

}

}'

{

"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[4gtV_OJESWWSgMnbTruKyA][personsearch][0]: SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]: SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]: SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]: SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]: SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }]",

"status": 400

}

Does the dynamic scripting for [mvel] disabled have something to do with it?

Thanks.

--

You received this message because you are subscribed to the Google Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


(Daniel Yim) #3

Thanks, Rafal. The *top_hits *aggregator is working now, but could you
explain why this is a security risk? We will need to use this feature in
production and now I am feeling uneasy about it.

On Thursday, July 24, 2014 10:28:32 AM UTC-5, Rafał Kuć wrote:

Hello!

The error is about scripting and dynamic scripting being disabled for
MVEL. The simplest way to make it work is turning on dynamic scripting by
adding script.disable_dynamic: false to your elasticsearch.yml file.
However for production it is not recommended for security reasons. You can
also try using scripting language that is sandboxed and allows for dynamic
scripting, like groovy.

*-- Regards, Rafał Kuć Performance Monitoring * Log Analytics * Search
Analytics Solr & Elasticsearch Support * *http://sematext.com/

Hello, I am trying to follow the field collapse example
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
on this page
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example,
which uses the new 1.3.0 top_hits aggregator to return the top scoring
document for a given criteria. To my knowledge, I set up the settings and
mappings correctly, but I am getting a strange error when I try to run the
query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"idx_analyzer": {
"tokenizer": "whitespace",
"filter": [
"lowercase",
"snowball",
"XYZSynFilter"
]
},
"sch_analyzer": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop"
]
},
"sch_comma_analyzer": {
"tokenizer": "CommaTokenizer",
"filter": [
"standard",
"lowercase",
"stop"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids virus, aids, retrovirology, hiv"
],
"expand": true,
"ignore_case": true
}
},
"tokenizer": {
"CommaTokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
},
"mappings": {
"employees": {
"properties": {
"fullName": {
"type": "string",
"search_analyzer": "sch_analyzer"
},
"specialty": {
"type": "string",
"search_analyzer": "sch_comma_analyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'
{
"fullName": "Don White",
"specialty": "Adult Retrovirology, aids, hiv"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'
{
"fullName": "Don White",
"specialty": "general practitioner, physician, general, primary care"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'
{
"fullName": "Don White",
"specialty": "icu, er"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'
{
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'
{
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'
{
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'
{
"fullName": "Carter Taylor",
"specialty": "neurosurgery, brain surgery, brain tumor"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'
{
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}'

Executing this search (per the example) gives me an error
curl -XGET "
http://localhost:9200/personsearch/employees/_search?pretty=true" -d'
{
"query": {
"query_string": {
"query": "icu"
}
},
"aggs": {
"most-rel-profile": {
"terms": {
"field": "profileName",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit": {
"max": {
"script": "_doc.score"
}
}
}
}
}
}'

{
"error": "SearchPhaseExecutionException[Failed to execute phase [query],
all shards failed; shardFailures
{[4gtV_OJESWWSgMnbTruKyA][personsearch][0]:
SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]:
SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]:
SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]:
SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]:
SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested:* ScriptException[dynamic scripting for
[mvel] disabled];* }]",
"status": 400
}

Does the dynamic scripting for [mvel] disabled have something to do
with it?

Thanks.

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Rafał Kuć) #4

Hello Daniel!

MVEL scripts can't be sandboxed and they provide a lot of features, however when running Elasticsearch with too much user rights one can exploit that situation, i.e. http://bouk.co/blog/elasticsearch-rce/. This resulted in dynamic scripting being disabled by default in Elasticsearch 1.2.0 (http://www.elasticsearch.org/blog/elasticsearch-1-2-0-released/) and enabled again in 1.3.0 (http://www.elasticsearch.org/blog/elasticsearch-1-3-0-released/) for languages like Groovy which can be sandboxed.

--

Regards,

Rafał Kuć

Performance Monitoring * Log Analytics * Search Analytics

Solr & Elasticsearch Support * http://sematext.com/

Thanks, Rafal. The top_hits aggregator is working now, but could you explain why this is a security risk? We will need to use this feature in production and now I am feeling uneasy about it.

On Thursday, July 24, 2014 10:28:32 AM UTC-5, Rafał Kuć wrote:

Hello!

The error is about scripting and dynamic scripting being disabled for MVEL. The simplest way to make it work is turning on dynamic scripting by adding script.disable_dynamic: false to your elasticsearch.yml file. However for production it is not recommended for security reasons. You can also try using scripting language that is sandboxed and allows for dynamic scripting, like groovy.

--

Regards,

Rafał Kuć

Performance Monitoring * Log Analytics * Search Analytics

Solr & Elasticsearch Support * http://sematext.com/

Hello, I am trying to follow the field collapse example on this page, which uses the new 1.3.0 top_hits aggregator to return the top scoring document for a given criteria. To my knowledge, I set up the settings and mappings correctly, but I am getting a strange error when I try to run the query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"

curl -XPUT "http://localhost:9200/personsearch" -d'

{

"settings": {

"index": {

 "analysis": {


   "analyzer": {


     "idx_analyzer": {


       "tokenizer": "whitespace",


       "filter": [


         "lowercase",


         "snowball",


         "XYZSynFilter"


       ]


     },


     "sch_analyzer": {


       "tokenizer": "standard",


       "filter": [


         "standard",


         "lowercase",


         "stop"


       ]


     },


     "sch_comma_analyzer": {


       "tokenizer": "CommaTokenizer",


       "filter": [


         "standard",


         "lowercase",


         "stop"


       ]


     }


   },


   "filter": {


     "XYZSynFilter": {


       "type": "synonym",


       "synonyms": [


         "aids virus, aids, retrovirology, hiv"


       ],


       "expand": true,


       "ignore_case": true


     }


   },


   "tokenizer": {


     "CommaTokenizer": {


       "type": "pattern",


       "pattern": ","


     }


   }


 }

}

},

"mappings": {

"employees": {

 "properties": {


   "fullName": {


     "type": "string",


     "search_analyzer": "sch_analyzer"


   },


   "specialty": {


     "type": "string",


     "search_analyzer": "sch_comma_analyzer"


   }


 }

}

}

}'

curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'

{

"fullName": "Don White",

"specialty": "Adult Retrovirology, aids, hiv"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'

{

"fullName": "Don White",

"specialty": "general practitioner, physician, general, primary care"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'

{

"fullName": "Don White",

"specialty": "icu, er"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'

{

"fullName": "Terrance Gartner",

"specialty": "oncology, cancer, research, tumor, polyp"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'

{

"fullName": "Terrance Gartner",

"specialty": "physician, general, GP, primary care, aids"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'

{

"fullName": "Terrance Gartner",

"specialty": "emergency care, icu, ambulance, er, urgent"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'

{

"fullName": "Carter Taylor",

"specialty": "neurosurgery, brain surgery, brain tumor"

}'

curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'

{

"fullName": "Carter Taylor",

"specialty": "trauma, icu, emergency care, ER, urgent care"

}'

Executing this search (per the example) gives me an error

curl -XGET "http://localhost:9200/personsearch/employees/_search?pretty=true" -d'

{

"query": {

"query_string": {

 "query": "icu"

}

},

"aggs": {

"most-rel-profile": {

 "terms": {


   "field": "profileName",


   "order": {


     "top_hit": "desc"


   }


 },


 "aggs": {


   "top_tags_hits": {


     "top_hits": {}


   },


   "top_hit": {


     "max": {


       "script": "_doc.score"


     }


   }


 }

}

}

}'

{

"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[4gtV_OJESWWSgMnbTruKyA][personsearch][0]: SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]: SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]: SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]: SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]: SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n "query": {\n "query_string": {\n "query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n "terms": {\n "field": "profileName",\n "order": {\n "top_hit": "desc"\n }\n },\n "aggs": {\n "top_tags_hits": {\n "top_hits": {}\n },\n "top_hit": {\n "max": {\n "script": "_doc.score"\n }\n }\n }\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled]; }]",

"status": 400

}

Does the dynamic scripting for [mvel] disabled have something to do with it?

Thanks.

--

You received this message because you are subscribed to the Google Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.--

You received this message because you are subscribed to the Google Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #5

You can switch to Lucene "expression" language to access score.

The example in the docs is a bit unfortunate, as it uses MVEL, which has to
be enabled in ES 1.3 explicitly.

Here is a modified example I made using the "expression" language:

Jörg

On Thu, Jul 24, 2014 at 6:22 PM, Daniel Yim danielyim@gmail.com wrote:

Thanks, Rafal. The *top_hits *aggregator is working now, but could you
explain why this is a security risk? We will need to use this feature in
production and now I am feeling uneasy about it.

On Thursday, July 24, 2014 10:28:32 AM UTC-5, Rafał Kuć wrote:

Hello!

The error is about scripting and dynamic scripting being disabled for
MVEL. The simplest way to make it work is turning on dynamic scripting by
adding script.disable_dynamic: false to your elasticsearch.yml file.
However for production it is not recommended for security reasons. You can
also try using scripting language that is sandboxed and allows for dynamic
scripting, like groovy.

*-- Regards, Rafał Kuć Performance Monitoring * Log Analytics * Search
Analytics Solr & Elasticsearch Support * *http://sematext.com/

Hello, I am trying to follow the field collapse example
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
on this page
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
, which uses the new 1.3.0 top_hits aggregator to return the top
scoring document for a given criteria. To my knowledge, I set up the
settings and mappings correctly, but I am getting a strange error when I
try to run the query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"idx_analyzer": {
"tokenizer": "whitespace",
"filter": [
"lowercase",
"snowball",
"XYZSynFilter"
]
},
"sch_analyzer": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop"
]
},
"sch_comma_analyzer": {
"tokenizer": "CommaTokenizer",
"filter": [
"standard",
"lowercase",
"stop"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids virus, aids, retrovirology, hiv"
],
"expand": true,
"ignore_case": true
}
},
"tokenizer": {
"CommaTokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
},
"mappings": {
"employees": {
"properties": {
"fullName": {
"type": "string",
"search_analyzer": "sch_analyzer"
},
"specialty": {
"type": "string",
"search_analyzer": "sch_comma_analyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'
{
"fullName": "Don White",
"specialty": "Adult Retrovirology, aids, hiv"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'
{
"fullName": "Don White",
"specialty": "general practitioner, physician, general, primary care"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'
{
"fullName": "Don White",
"specialty": "icu, er"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'
{
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'
{
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'
{
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'
{
"fullName": "Carter Taylor",
"specialty": "neurosurgery, brain surgery, brain tumor"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'
{
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}'

Executing this search (per the example) gives me an error
curl -XGET "http://localhost:9200/personsearch/employees/_
search?pretty=true" -d'
{
"query": {
"query_string": {
"query": "icu"
}
},
"aggs": {
"most-rel-profile": {
"terms": {
"field": "profileName",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit": {
"max": {
"script": "_doc.score"
}
}
}
}
}
}'

{
"error": "SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed; shardFailures {[4gtV_OJESWWSgMnbTruKyA][personsearch][0]:
SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]:
SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]:
SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]:
SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]:
SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested:* ScriptException[dynamic scripting for
[mvel] disabled];* }]",
"status": 400
}

Does the dynamic scripting for [mvel] disabled have something to do
with it?

Thanks.

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-
78a4-4419-a87d-e39ec3148fcf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFhAzKExHojqtOD%3D9R-vZCOjuY8UfQdu0FJ8Z1rUjwN_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Daniel Yim) #6

That gives me the following error:

{
"error": "SearchPhaseExecutionException[Failed to execute phase [query],
all shards failed; shardFailures
{[mzcfN2MzQjSKlWPHndRROA][personsearch][0]:
SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n "match":
{\n "_all": "icu"\n }\n },\n "aggs": {\n
"most-rel-agg": {\n "terms": {\n "field":
"fullName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "most-rel-profile": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "lang" : "expression",\n "script":
"_score"\n }\n }\n }\n }\n }\n}\n]]]; nested:
NoClassDefFoundError[org/antlr/runtime/RecognitionException]; nested:
ClassNotFoundException[org.antlr.runtime.RecognitionException];
}{[mzcfN2MzQjSKlWPHndRROA][personsearch][1]:
SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n "match":
{\n "_all": "icu"\n }\n },\n "aggs": {\n
"most-rel-agg": {\n "terms": {\n "field":
"fullName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "most-rel-profile": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "lang" : "expression",\n "script":
"_score"\n }\n }\n }\n }\n }\n}\n]]]; nested:
NoClassDefFoundError[org/antlr/runtime/RecognitionException]; nested:
ClassNotFoundException[org.antlr.runtime.RecognitionException];
}{[mzcfN2MzQjSKlWPHndRROA][personsearch][2]:
SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n "match":
{\n "_all": "icu"\n }\n },\n "aggs": {\n
"most-rel-agg": {\n "terms": {\n "field":
"fullName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "most-rel-profile": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "lang" : "expression",\n "script":
"_score"\n }\n }\n }\n }\n }\n}\n]]]; nested:
NoClassDefFoundError[org/antlr/runtime/RecognitionException]; nested:
ClassNotFoundException[org.antlr.runtime.RecognitionException];
}{[mzcfN2MzQjSKlWPHndRROA][personsearch][3]:
SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n "match":
{\n "_all": "icu"\n }\n },\n "aggs": {\n
"most-rel-agg": {\n "terms": {\n "field":
"fullName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "most-rel-profile": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "lang" : "expression",\n "script":
"_score"\n }\n }\n }\n }\n }\n}\n]]]; nested:
NoClassDefFoundError[org/antlr/runtime/RecognitionException]; nested:
ClassNotFoundException[org.antlr.runtime.RecognitionException];
}{[mzcfN2MzQjSKlWPHndRROA][personsearch][4]:
SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n "match":
{\n "_all": "icu"\n }\n },\n "aggs": {\n
"most-rel-agg": {\n "terms": {\n "field":
"fullName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "most-rel-profile": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "lang" : "expression",\n "script":
"_score"\n }\n }\n }\n }\n }\n}\n]]]; nested:
NoClassDefFoundError[org/antlr/runtime/RecognitionException]; nested:
ClassNotFoundException[org.antlr.runtime.RecognitionException]; }]",
"status": 400
}

On Thursday, July 24, 2014 12:05:17 PM UTC-5, Jörg Prante wrote:

You can switch to Lucene "expression" language to access score.

The example in the docs is a bit unfortunate, as it uses MVEL, which has
to be enabled in ES 1.3 explicitly.

Here is a modified example I made using the "expression" language:

https://gist.github.com/jprante/7a1763c7073f2b8084a6

Jörg

On Thu, Jul 24, 2014 at 6:22 PM, Daniel Yim <dani...@gmail.com
<javascript:>> wrote:

Thanks, Rafal. The *top_hits *aggregator is working now, but could you
explain why this is a security risk? We will need to use this feature in
production and now I am feeling uneasy about it.

On Thursday, July 24, 2014 10:28:32 AM UTC-5, Rafał Kuć wrote:

Hello!

The error is about scripting and dynamic scripting being disabled for
MVEL. The simplest way to make it work is turning on dynamic scripting by
adding script.disable_dynamic: false to your elasticsearch.yml file.
However for production it is not recommended for security reasons. You can
also try using scripting language that is sandboxed and allows for dynamic
scripting, like groovy.

*-- Regards, Rafał Kuć Performance Monitoring * Log Analytics * Search
Analytics Solr & Elasticsearch Support * *http://sematext.com/

Hello, I am trying to follow the field collapse example
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
on this page
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
, which uses the new 1.3.0 top_hits aggregator to return the top
scoring document for a given criteria. To my knowledge, I set up the
settings and mappings correctly, but I am getting a strange error when I
try to run the query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"idx_analyzer": {
"tokenizer": "whitespace",
"filter": [
"lowercase",
"snowball",
"XYZSynFilter"
]
},
"sch_analyzer": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop"
]
},
"sch_comma_analyzer": {
"tokenizer": "CommaTokenizer",
"filter": [
"standard",
"lowercase",
"stop"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids virus, aids, retrovirology, hiv"
],
"expand": true,
"ignore_case": true
}
},
"tokenizer": {
"CommaTokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
},
"mappings": {
"employees": {
"properties": {
"fullName": {
"type": "string",
"search_analyzer": "sch_analyzer"
},
"specialty": {
"type": "string",
"search_analyzer": "sch_comma_analyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'
{
"fullName": "Don White",
"specialty": "Adult Retrovirology, aids, hiv"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'
{
"fullName": "Don White",
"specialty": "general practitioner, physician, general, primary care"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'
{
"fullName": "Don White",
"specialty": "icu, er"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'
{
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'
{
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'
{
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'
{
"fullName": "Carter Taylor",
"specialty": "neurosurgery, brain surgery, brain tumor"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'
{
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}'

Executing this search (per the example) gives me an error
curl -XGET "http://localhost:9200/personsearch/employees/_
search?pretty=true" -d'
{
"query": {
"query_string": {
"query": "icu"
}
},
"aggs": {
"most-rel-profile": {
"terms": {
"field": "profileName",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit": {
"max": {
"script": "_doc.score"
}
}
}
}
}
}'

{
"error": "SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed; shardFailures {[4gtV_OJESWWSgMnbTruKyA][personsearch][0]:
SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]:
SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]:
SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]:
SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]:
SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested:* ScriptException[dynamic scripting for
[mvel] disabled];* }]",
"status": 400
}

Does the dynamic scripting for [mvel] disabled have something to do
with it?

Thanks.

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-
78a4-4419-a87d-e39ec3148fcf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/acf4f717-359d-461c-8bb8-a65e7f4cfb08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Daniel Yim) #7

Also, I was hoping that this example will give me the entire contents of
the top scoring document, not just its score.

Here is the result I get when running this query:
https://gist.github.com/384d88ee0015009033d7.git

{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 5,
"max_score": 0.3125,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "6",
"_score": 0.3125,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "3",
"_score": 0.2972674,
"_source": {
"fullName": "Don White",
"specialty": "icu, er"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "8",
"_score": 0.18579213,
"_source": {
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "4",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp, icu"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "5",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids,
icu"
}
}
]
},
"aggregations": {
"most-rel-agg": {
"buckets": [
{
"key": "terrance gartner",
"doc_count": 3,
"most-rel-profile": {
"hits": {
"total": 3,
"max_score": 0.3125,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "6",
"_score": 0.3125,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance,
er, urgent"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "4",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research,
tumor, polyp, icu"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "5",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary
care, aids, icu"
}
}
]
}
},
"top_hit": {
"value": 0.3125
}
},
{
"key": "don white",
"doc_count": 1,
"most-rel-profile": {
"hits": {
"total": 1,
"max_score": 0.2972674,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "3",
"_score": 0.2972674,
"_source": {
"fullName": "Don White",
"specialty": "icu, er"
}
}
]
}
},
"top_hit": {
"value": 0.2972674071788788
}
},
{
"key": "carter taylor",
"doc_count": 1,
"most-rel-profile": {
"hits": {
"total": 1,
"max_score": 0.18579213,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "8",
"_score": 0.18579213,
"_source": {
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care,
ER, urgent care"
}
}
]
}
},
"top_hit": {
"value": 0.18579213321208954
}
}
]
}
}
}

On Thursday, July 24, 2014 12:05:17 PM UTC-5, Jörg Prante wrote:

You can switch to Lucene "expression" language to access score.

The example in the docs is a bit unfortunate, as it uses MVEL, which has
to be enabled in ES 1.3 explicitly.

Here is a modified example I made using the "expression" language:

https://gist.github.com/jprante/7a1763c7073f2b8084a6

Jörg

On Thu, Jul 24, 2014 at 6:22 PM, Daniel Yim <dani...@gmail.com
<javascript:>> wrote:

Thanks, Rafal. The *top_hits *aggregator is working now, but could you
explain why this is a security risk? We will need to use this feature in
production and now I am feeling uneasy about it.

On Thursday, July 24, 2014 10:28:32 AM UTC-5, Rafał Kuć wrote:

Hello!

The error is about scripting and dynamic scripting being disabled for
MVEL. The simplest way to make it work is turning on dynamic scripting by
adding script.disable_dynamic: false to your elasticsearch.yml file.
However for production it is not recommended for security reasons. You can
also try using scripting language that is sandboxed and allows for dynamic
scripting, like groovy.

*-- Regards, Rafał Kuć Performance Monitoring * Log Analytics * Search
Analytics Solr & Elasticsearch Support * *http://sematext.com/

Hello, I am trying to follow the field collapse example
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
on this page
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
, which uses the new 1.3.0 top_hits aggregator to return the top
scoring document for a given criteria. To my knowledge, I set up the
settings and mappings correctly, but I am getting a strange error when I
try to run the query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"idx_analyzer": {
"tokenizer": "whitespace",
"filter": [
"lowercase",
"snowball",
"XYZSynFilter"
]
},
"sch_analyzer": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop"
]
},
"sch_comma_analyzer": {
"tokenizer": "CommaTokenizer",
"filter": [
"standard",
"lowercase",
"stop"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids virus, aids, retrovirology, hiv"
],
"expand": true,
"ignore_case": true
}
},
"tokenizer": {
"CommaTokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
},
"mappings": {
"employees": {
"properties": {
"fullName": {
"type": "string",
"search_analyzer": "sch_analyzer"
},
"specialty": {
"type": "string",
"search_analyzer": "sch_comma_analyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'
{
"fullName": "Don White",
"specialty": "Adult Retrovirology, aids, hiv"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'
{
"fullName": "Don White",
"specialty": "general practitioner, physician, general, primary care"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'
{
"fullName": "Don White",
"specialty": "icu, er"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'
{
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'
{
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'
{
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'
{
"fullName": "Carter Taylor",
"specialty": "neurosurgery, brain surgery, brain tumor"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'
{
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}'

Executing this search (per the example) gives me an error
curl -XGET "http://localhost:9200/personsearch/employees/_
search?pretty=true" -d'
{
"query": {
"query_string": {
"query": "icu"
}
},
"aggs": {
"most-rel-profile": {
"terms": {
"field": "profileName",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit": {
"max": {
"script": "_doc.score"
}
}
}
}
}
}'

{
"error": "SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed; shardFailures {[4gtV_OJESWWSgMnbTruKyA][personsearch][0]:
SearchParseException[[personsearch][0]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]:
SearchParseException[[personsearch][1]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]:
SearchParseException[[personsearch][2]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]:
SearchParseException[[personsearch][3]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested: ScriptException[dynamic scripting for
[mvel] disabled]; }{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]:
SearchParseException[[personsearch][4]: query[_all:icu],from[-1],size[-1]:
Parse Failure [Failed to parse source [{\n "query": {\n
"query_string": {\n "query": "icu"\n }\n },\n "aggs":
{\n "most-rel-profile": {\n "terms": {\n "field":
"profileName",\n "order": {\n "top_hit": "desc"\n
}\n },\n "aggs": {\n "top_tags_hits": {\n
"top_hits": {}\n },\n "top_hit": {\n "max":
{\n "script": "_doc.score"\n }\n }\n
}\n }\n }\n}\n]]]; nested:* ScriptException[dynamic scripting for
[mvel] disabled];* }]",
"status": 400
}

Does the dynamic scripting for [mvel] disabled have something to do
with it?

Thanks.

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-
78a4-4419-a87d-e39ec3148fcf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7675dfb7-66e4-4626-9fa3-01ec4a5494bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #8

The ANTLR class error is not related , I have no ANTLR here, it must be an
erro at your side.

Not sure where you miss the entire content - I see the aggregated docs with
all the _source under "most-rel-profile", ordered by score descending.

Jörg

On Thu, Jul 24, 2014 at 8:52 PM, Daniel Yim danielyim@gmail.com wrote:

Also, I was hoping that this example will give me the entire contents of
the top scoring document, not just its score.

Here is the result I get when running this query:
https://gist.github.com/384d88ee0015009033d7.git

{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 5,
"max_score": 0.3125,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "6",
"_score": 0.3125,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "3",
"_score": 0.2972674,
"_source": {
"fullName": "Don White",
"specialty": "icu, er"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "8",
"_score": 0.18579213,
"_source": {
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "4",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp, icu"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "5",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids,
icu"
}
}
]
},
"aggregations": {
"most-rel-agg": {
"buckets": [
{
"key": "terrance gartner",
"doc_count": 3,
"most-rel-profile": {
"hits": {
"total": 3,
"max_score": 0.3125,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "6",
"_score": 0.3125,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu,
ambulance, er, urgent"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "4",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research,
tumor, polyp, icu"
}
},
{
"_index": "personsearch",
"_type": "employees",
"_id": "5",
"_score": 0.095891505,
"_source": {
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP,
primary care, aids, icu"
}
}
]
}
},
"top_hit": {
"value": 0.3125
}
},
{
"key": "don white",
"doc_count": 1,
"most-rel-profile": {
"hits": {
"total": 1,
"max_score": 0.2972674,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "3",
"_score": 0.2972674,
"_source": {
"fullName": "Don White",
"specialty": "icu, er"
}
}
]
}
},
"top_hit": {
"value": 0.2972674071788788
}
},
{
"key": "carter taylor",
"doc_count": 1,
"most-rel-profile": {
"hits": {
"total": 1,
"max_score": 0.18579213,
"hits": [
{
"_index": "personsearch",
"_type": "employees",
"_id": "8",
"_score": 0.18579213,
"_source": {
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care,
ER, urgent care"
}
}
]
}
},
"top_hit": {
"value": 0.18579213321208954
}
}
]
}
}
}

On Thursday, July 24, 2014 12:05:17 PM UTC-5, Jörg Prante wrote:

You can switch to Lucene "expression" language to access score.

The example in the docs is a bit unfortunate, as it uses MVEL, which has
to be enabled in ES 1.3 explicitly.

Here is a modified example I made using the "expression" language:

https://gist.github.com/jprante/7a1763c7073f2b8084a6

Jörg

On Thu, Jul 24, 2014 at 6:22 PM, Daniel Yim dani...@gmail.com wrote:

Thanks, Rafal. The *top_hits *aggregator is working now, but could you
explain why this is a security risk? We will need to use this feature in
production and now I am feeling uneasy about it.

On Thursday, July 24, 2014 10:28:32 AM UTC-5, Rafał Kuć wrote:

Hello!

The error is about scripting and dynamic scripting being disabled for
MVEL. The simplest way to make it work is turning on dynamic scripting by
adding script.disable_dynamic: false to your elasticsearch.yml file.
However for production it is not recommended for security reasons. You can
also try using scripting language that is sandboxed and allows for dynamic
scripting, like groovy.

*-- Regards, Rafał Kuć Performance Monitoring * Log Analytics * Search
Analytics Solr & Elasticsearch Support * *http://sematext.com/

Hello, I am trying to follow the field collapse example
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
on this page
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#_field_collapse_example
, which uses the new 1.3.0 top_hits aggregator to return the top
scoring document for a given criteria. To my knowledge, I set up the
settings and mappings correctly, but I am getting a strange error when I
try to run the query that the example provided.

curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"idx_analyzer": {
"tokenizer": "whitespace",
"filter": [
"lowercase",
"snowball",
"XYZSynFilter"
]
},
"sch_analyzer": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop"
]
},
"sch_comma_analyzer": {
"tokenizer": "CommaTokenizer",
"filter": [
"standard",
"lowercase",
"stop"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids virus, aids, retrovirology, hiv"
],
"expand": true,
"ignore_case": true
}
},
"tokenizer": {
"CommaTokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
},
"mappings": {
"employees": {
"properties": {
"fullName": {
"type": "string",
"search_analyzer": "sch_analyzer"
},
"specialty": {
"type": "string",
"search_analyzer": "sch_comma_analyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/employees/1" -d'
{
"fullName": "Don White",
"specialty": "Adult Retrovirology, aids, hiv"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/2" -d'
{
"fullName": "Don White",
"specialty": "general practitioner, physician, general, primary care"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/3" -d'
{
"fullName": "Don White",
"specialty": "icu, er"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/4" -d'
{
"fullName": "Terrance Gartner",
"specialty": "oncology, cancer, research, tumor, polyp"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/5" -d'
{
"fullName": "Terrance Gartner",
"specialty": "physician, general, GP, primary care, aids"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/6" -d'
{
"fullName": "Terrance Gartner",
"specialty": "emergency care, icu, ambulance, er, urgent"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/7" -d'
{
"fullName": "Carter Taylor",
"specialty": "neurosurgery, brain surgery, brain tumor"
}'
curl -XPUT "http://localhost:9200/personsearch/employees/8" -d'
{
"fullName": "Carter Taylor",
"specialty": "trauma, icu, emergency care, ER, urgent care"
}'

Executing this search (per the example) gives me an error
curl -XGET "http://localhost:9200/personsearch/employees/_search?
pretty=true" -d'
{
"query": {
"query_string": {
"query": "icu"
}
},
"aggs": {
"most-rel-profile": {
"terms": {
"field": "profileName",
"order": {
"top_hit": "desc"
}
},
"aggs": {
"top_tags_hits": {
"top_hits": {}
},
"top_hit": {
"max": {
"script": "_doc.score"
}
}
}
}
}
}'

{
"error": "SearchPhaseExecutionException[Failed to execute phase
[query], all shards failed; shardFailures {[4gtV_OJESWWSgMnbTruKyA][
personsearch][0]: SearchParseException[[personsearch][0]:
query[_all:icu],from[-1],size[-1]: Parse Failure [Failed to parse
source [{\n "query": {\n "query_string": {\n "query":
"icu"\n }\n },\n "aggs": {\n "most-rel-profile": {\n
"terms": {\n "field": "profileName",\n "order": {\n
"top_hit": "desc"\n }\n },\n "aggs": {\n
"top_tags_hits": {\n "top_hits": {}\n },\n
"top_hit": {\n "max": {\n "script":
"_doc.score"\n }\n }\n }\n }\n }\n}\n]]];
nested: ScriptException[dynamic scripting for [mvel] disabled];
}{[4gtV_OJESWWSgMnbTruKyA][personsearch][1]: SearchParseException[[
personsearch][1]: query[_all:icu],from[-1],size[-1]: Parse Failure
[Failed to parse source [{\n "query": {\n "query_string": {\n
"query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile":
{\n "terms": {\n "field": "profileName",\n
"order": {\n "top_hit": "desc"\n }\n },\n
"aggs": {\n "top_tags_hits": {\n "top_hits": {}\n
},\n "top_hit": {\n "max": {\n
"script": "_doc.score"\n }\n }\n }\n }\n
}\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled];
}{[4gtV_OJESWWSgMnbTruKyA][personsearch][2]: SearchParseException[[
personsearch][2]: query[_all:icu],from[-1],size[-1]: Parse Failure
[Failed to parse source [{\n "query": {\n "query_string": {\n
"query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile":
{\n "terms": {\n "field": "profileName",\n
"order": {\n "top_hit": "desc"\n }\n },\n
"aggs": {\n "top_tags_hits": {\n "top_hits": {}\n
},\n "top_hit": {\n "max": {\n
"script": "_doc.score"\n }\n }\n }\n }\n
}\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled];
}{[4gtV_OJESWWSgMnbTruKyA][personsearch][3]: SearchParseException[[
personsearch][3]: query[_all:icu],from[-1],size[-1]: Parse Failure
[Failed to parse source [{\n "query": {\n "query_string": {\n
"query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile":
{\n "terms": {\n "field": "profileName",\n
"order": {\n "top_hit": "desc"\n }\n },\n
"aggs": {\n "top_tags_hits": {\n "top_hits": {}\n
},\n "top_hit": {\n "max": {\n
"script": "_doc.score"\n }\n }\n }\n }\n
}\n}\n]]]; nested: ScriptException[dynamic scripting for [mvel] disabled];
}{[4gtV_OJESWWSgMnbTruKyA][personsearch][4]: SearchParseException[[
personsearch][4]: query[_all:icu],from[-1],size[-1]: Parse Failure
[Failed to parse source [{\n "query": {\n "query_string": {\n
"query": "icu"\n }\n },\n "aggs": {\n "most-rel-profile":
{\n "terms": {\n "field": "profileName",\n
"order": {\n "top_hit": "desc"\n }\n },\n
"aggs": {\n "top_tags_hits": {\n "top_hits": {}\n
},\n "top_hit": {\n "max": {\n
"script": "_doc.score"\n }\n }\n }\n }\n
}\n}\n]]]; nested:* ScriptException[dynamic scripting for [mvel]
disabled];* }]",
"status": 400
}

Does the dynamic scripting for [mvel] disabled have something to do
with it?

Thanks.

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a
4-4419-a87d-e39ec3148fcf%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/79c1fff8-78a4-4419-a87d-e39ec3148fcf%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/43346d99-6385-4a24-bd8e-8e7482f39a36%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7675dfb7-66e4-4626-9fa3-01ec4a5494bb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7675dfb7-66e4-4626-9fa3-01ec4a5494bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG86y0Hm0kJHdH-8qxHKjWF%2B0p%3DP7R554t5ZN98giBJGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #9