Accessing a scripted field in an elasticsearch match query

Elastic Stack Elasticsearch
#1

I have a query like the following

{
  "script_fields": {
    "my_script_field": {
      "script": "..."
    }
  },
  "query": {
    "match": {
      "my_script_field":"*"
    }
  }
}

and it returns empty.

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}

I can see that the script is actually working by simply retuning everything
like:

{
  "fields": [
  "_source"
  ],
  "script_fields": {
    "my_script_field": {
      "script": "..."
    }
  },
  "query": {
    match_all: {}
  }
}

Example results:

  "hits": {
    "total": 1008681,
    "max_score": 1,
    "hits": [
      {
        "_index": "logstash-2016.08.27",
        "_type": "traffic",
        "_id": "AVbLDW8qw2vffjMOfTxb",
        "_score": 1,
        "_source": {
          "@version": "1",
          "@timestamp": "2016-08-27T06:11:46.000Z",
           .................
          }
        },
        "fields": {
          "my_scripted_field": [
            "Asia"
          ]
        }

but I cant seem to access it in the query context in any other way.

my question so is:

  • Where is the script output saved in memory? (_fields?)
  • How to access it via a query? (without copying the script over to a filter script)

Thanks!

#2

script_fields are to create custom fields that get returned out of the documents that match your query. You cannot query them, so effectively your query queries a non existing field hence it returns no documents.

Can you elaborate on what your documents look like and how you would like to query them? We may find a a way to help then. script_fields may be a way, but maybe there are others.

#3

Were working on an analytics framework based on ELK.
We have many types of logs ranging from simple firewall logs (like the one in the original question), to more complex logs like windows services (dlls paths, service names etc..)

One of the use cases for our analysts is to create and interact transparently with scripted fields.

For example:
a script that will extract the containing directory of the dll (instead of full path), and than group by directories.
The flow would be:

  • user creates a directory field
  • user asks for aggregation on that field (count for example)
  • the framework would than convert the script_field to a scripted aggregation (it returns only aggregated results OR hits, never both in the same query)
  • internally we keep track of the field name and script that the user asked for and do the before querying elasticsearch.

We wanted to implement a similar feature for the hits (user creates directory field, than wants to match a specific directory).
I guess we could have done something similar with scripted filter, but I wished to avoid it because, we rely on the lucene query string for simple filters (and wrapper functions for aggregations).

Would love to hear your thoughts :slight_smile:

Thanks again!

#4