Scripted field - "no field found" on field presence check

[philhagen]

With a painless scripted field, the recommended field presence logic itself causes a "No field found" error, but not on all records - in fact just four of ~930k fail with the error (below).

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 73,
    "successful": 72,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "netflow-2018.07.06",
        "node": "biwB6-HEQCC5kTcwa4AAEw",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "if ( doc['destination_geo.asn'].size() == 0 || doc['destination_geo.as_org.keyword'].size() == 0 ) { ",
            "         ^---- HERE"
          ],
          "script": "if ( doc['destination_geo.asn'].size() == 0 || doc['destination_geo.as_org.keyword'].size() == 0 ) { return 'ASN: Not Available' } else { return 'ASN' + doc['destination_geo.asn'].value + ': ' + doc['destination_geo.as_org.keyword'].value }",
          "lang": "painless",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "No field found for [destination_geo.asn] in mapping with types []"
          }
        }
      }
    ]
  },
  "hits": {
    "total": 4,
    "max_score": null,
    "hits": []
  }
}

However, for many other other documents that do not have a destination_geo.asn field, the scripted field as stated above works properly.

I experimented with using doc.containsKey() as indicated here. The four failing records work fine with this function but all other records missing a destination_geo.asn field fail.

[philhagen]

I should add that this was tested with 7.5.1 and 7.6.1 - same behavior.

[stu (Stuart Tettemer)]

This sounds like you're missing a mapping for destination_geo.asn in one of your indices. Is that the case?

When there's a mapping for a field, doc['destination_geo.asn'].size() == 0 is the best way to check if a value is missing for the field in a document.

If there is no mapping for a field, things are trickier, you have to check !doc.containsKey('destination_geo.asn') before checking doc['destination_geo.asn'].size() == 0.

The mapping ensures the destination_geo.asn exists in doc.

To put it together, try the following:

if (!doc.containsKey('destination_geo.asn') || doc['destination_geo.asn'].size() == 0 ....
If that doesn't work, it'd help to see your mappings and more of your script.

[philhagen]

Hi - thanks for the assist. This is bizarre....

index netflow-2018.07.06 does not have the mapping but other indices do. These were all created during the same ingest process.

$ curl -S -XGET http://127.0.0.1:9200/netflow-2018.07.06/_mapping | jq '.'
{
  "netflow-2018.07.06": {
    "mappings": {
      "dynamic_templates": [
        {
          "ip_address_fields": {
            "match": "*_ip",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "norms": false,
              "type": "text"
            }
          }
        }
      ],
      "properties": {
...removed...
        "destination_geo": {
          "dynamic": "true",
          "properties": {
            "latitude": {
              "type": "half_float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "half_float"
            }
          }
        },
...removed...
}

However:

$ curl -S -XGET http://127.0.0.1:9200/netflow-2018.07.05/_mapping | jq '.'
{
...removed...
        "destination_geo": {
          "dynamic": "true",
          "properties": {
            "as_org": {
              "type": "text",
              "norms": false,
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "asn": {
              "type": "long"
            },
...removed...
}

[philhagen]

I used the dual verification syntax with both !doc.containsKey('x') and doc['x'].size() == 0 and it worked ok. however, I'm pretty puzzled as to why the mapping was not applied to the adjacent indices.