Scripted field - "no field found" on field presence check

With a painless scripted field, the recommended field presence logic itself causes a "No field found" error, but not on all records - in fact just four of ~930k fail with the error (below).

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 73,
    "successful": 72,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "netflow-2018.07.06",
        "node": "biwB6-HEQCC5kTcwa4AAEw",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "if ( doc['destination_geo.asn'].size() == 0 || doc['destination_geo.as_org.keyword'].size() == 0 ) { ",
            "         ^---- HERE"
          ],
          "script": "if ( doc['destination_geo.asn'].size() == 0 || doc['destination_geo.as_org.keyword'].size() == 0 ) { return 'ASN: Not Available' } else { return 'ASN' + doc['destination_geo.asn'].value + ': ' + doc['destination_geo.as_org.keyword'].value }",
          "lang": "painless",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "No field found for [destination_geo.asn] in mapping with types []"
          }
        }
      }
    ]
  },
  "hits": {
    "total": 4,
    "max_score": null,
    "hits": []
  }
}

However, for many other other documents that do not have a destination_geo.asn field, the scripted field as stated above works properly.

I experimented with using doc.containsKey() as indicated here. The four failing records work fine with this function but all other records missing a destination_geo.asn field fail.

I should add that this was tested with 7.5.1 and 7.6.1 - same behavior.

This sounds like you're missing a mapping for destination_geo.asn in one of your indices. Is that the case?

When there's a mapping for a field, doc['destination_geo.asn'].size() == 0 is the best way to check if a value is missing for the field in a document.

If there is no mapping for a field, things are trickier, you have to check !doc.containsKey('destination_geo.asn') before checking doc['destination_geo.asn'].size() == 0.

The mapping ensures the destination_geo.asn exists in doc.

To put it together, try the following:

if (!doc.containsKey('destination_geo.asn') || doc['destination_geo.asn'].size() == 0 ....
If that doesn't work, it'd help to see your mappings and more of your script.

Hi - thanks for the assist. This is bizarre....

index netflow-2018.07.06 does not have the mapping but other indices do. These were all created during the same ingest process.

$ curl -S -XGET http://127.0.0.1:9200/netflow-2018.07.06/_mapping | jq '.'
{
  "netflow-2018.07.06": {
    "mappings": {
      "dynamic_templates": [
        {
          "ip_address_fields": {
            "match": "*_ip",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "norms": false,
              "type": "text"
            }
          }
        }
      ],
      "properties": {
...removed...
        "destination_geo": {
          "dynamic": "true",
          "properties": {
            "latitude": {
              "type": "half_float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "half_float"
            }
          }
        },
...removed...
}

However:

$ curl -S -XGET http://127.0.0.1:9200/netflow-2018.07.05/_mapping | jq '.'
{
...removed...
        "destination_geo": {
          "dynamic": "true",
          "properties": {
            "as_org": {
              "type": "text",
              "norms": false,
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "asn": {
              "type": "long"
            },
...removed...
}

I used the dual verification syntax with both !doc.containsKey('x') and doc['x'].size() == 0 and it worked ok. however, I'm pretty puzzled as to why the mapping was not applied to the adjacent indices.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.