_source variables in post_filter

[Brian_Keith]

I was using ES 5.4 and upgraded to 6.3. In 5.4 using a painless script in a post_filter I was able to access array data from a document like def lists = params._source['lists'];. Now in 6.3 I don't seem to be able to access the data.

PUT my_index/_doc/1
{
  "message": "some arrays in this document...",
  "tags":  [ "elasticsearch", "wow" ], 
  "lists": [ 
    {
      "name": "prog_list",
      "description": "programming list"
    },
    {
      "name": "cool_list",
      "description": "cool stuff list"
    }
  ]
}

PUT my_index/_doc/2 
{
  "message": "no arrays in this document...",
  "tags":  "elasticsearch",
  "lists": {
    "name": "prog_list 2",
    "description": "programming list 2"
  }
}

GET my_index/_search
{
  "query": {
    "match": {
      "tags": "elasticsearch" 
    }
  },
  "stored_fields": [
    "_source"
  ],
  "post_filter": {
    "script": {
      "script": {
        "source": """
            def lists = params._source.lists ;
            if (lists[0].name === 'prog_list') {
              return true;
            }
            return false;
        """
      }
    }
  }
}

Result is

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 3,
    "skipped": 0,
    "failed": 2,
    "failures": [
      {
        "shard": 2,
        "index": "my_index",
        "node": "mDf0dL9ISkSrEqNjWe2BFQ",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "lists = params._source['lists'];\n            ",
            "              ^---- HERE"
          ],
          "script": """
            def lists = params._source['lists'];
            if (lists[0].name === 'prog_list') {
              return true;
            }
            return false;
""",
          "lang": "painless",
          "caused_by": {
            "type": "null_pointer_exception",
            "reason": null
          }
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}

I tried using doc['lists'] and got an No field found for [lists] in mapping with types []

Is there anyway to access that array?

[rjernst (Ryan Ernst)]

Access to _source was intentionally removed in filter scripts as it is incredibly trappy due to the orders of magnitude performance difference compared to using doc values.

Have you considered denormalizing your inner objects? Or using an inner hits query?