Painless scripting field question

Elastic Stack Elasticsearch
1

I have a number of document, each document will have a src_ip field.
Some of the documents MIGHT also have a srcip_host field.

I would like to have scripted field, type is string(text), that takes the srcip_host value if that exists, otherwise just take the src_ip field.

The mapping of the src_ip and srcip_host are as follows:

"srcip_host": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
"srcip": {
            "type": "ip"
          },

In my Index_pattern, the scripted field is as following:

src_host_or_ip	painless	if (doc["srcip_host.keyword"].value != null) return doc["srcip_host.keyword"].value; else return doc["srcip"].value

But I got the following runtime warning from kibana web page.
Courier Fetch: 5 of 50 shards failed.

By investigating more, I found the following runtime error from kibana web client:

Object
index
:
"logstash-2017.07.17"
node
:
"iRl6kRKrTSGWqxFYI9i0rQ"
reason
:
Object
caused_by
:
Object
lang
:
"painless"
reason
:
"runtime error"
script
:
"if (doc['srcip_host.keyword'].value != null) return doc['srcip_host.keyword'].value;↵else return doc['srcip'].value"
script_stack
:
Array(3)
0
:
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:80)"
1
:
"if (doc['srcip_host.keyword'].value != null) "
2
:
"        ^---- HERE"
length
:
3```

The ^---HERE seems to pointing to doc['srcip_host.keyword'].value, is that because the field might not exist for certain documents?

How to deal with this type of problems?

Thanks
2

It's a little difficult to read your example. The following does work:

PUT /foo
{
  "mappings": {
    "bar": {
      "properties": {
        "srcip": {
          "type": "ip"
        }
      }
    }
  }
}

PUT /foo/bar/1
{
  "srcip": "10.0.0.1"
}

PUT /foo/bar/2
{
  "srcip_host": "eskibars.com"
}

GET /foo/_search
{
  "script_fields": {
    "src_host_or_ip": {
      "script": {
        "lang": "painless",
        "inline": "if (doc['srcip_host.keyword'].value != null) { return doc['srcip_host.keyword'].value } else { return doc['srcip'].value }"
      }
    }
  }
}

Can you maybe format an example doc/query or at least the example response?

3

The problem is that it failed in some shards, not all of the shards. So it works for this simple documents set does not mean it will work for large number of documents.

I got the error from kibana side:
Courier Fetch: 5 of 50 shards failed.

The error is happening on the elasticsearch side in this file:
./core/src/main/java/org/elasticsearch/search/lookup/LeafDocLookup.java

    @Override
    public ScriptDocValues<?> get(Object key) {
        // assume its a string...
        String fieldName = key.toString();
        ScriptDocValues<?> scriptValues = localCacheFieldData.get(fieldName);
        if (scriptValues == null) {
            final MappedFieldType fieldType = mapperService.fullName(fieldName);
            if (fieldType == null) {
                throw new IllegalArgumentException("No field found for [" + fieldName + "] in mapping with types " + Arrays.toString(types));
            }

The fieldType is null. I do not understand the internals of the elastic search, the question is when the mapperService is populated for the shard. Is it shared across the shards or it is per shard thing?

Thanks

4

Do you define your mappings up front, or rely on dynamic mappings? I could see this happening if you rolled over to a new day's index, and had not yet seen any documents which contained srcip_host. You can protect against that by checking the doc for the field before trying to access it, with doc.containsKey('srcip_host.keyword').

5

I will try this, Thanks for the reply.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.