Using field or field.keyword errors

[pmarques (Paulo Marques)]

Hello all,

Could you help me decipher this? The issue is the use of field/field.keyword

Not sure where to start... Well ES 7.3

Note:
There are some "-------------- >>>" marks in this message.
I hope you find it

Documents look like:

{ "province": "British Columbia", "q_1": "Yes" }
{ "province": "British Columbia", "q_1": "No" }
{ "province": "British Columbia", "q_1": "Maybe" }
{ "province": "Alberta", "q_1": "Yes" }
{ "province": "Alberta", "q_1": "Maybe" }

The mapping is:

"province" : {
      "type" : "text",
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "ignore_above" : 256
        }
      }
    },
    "q_1" : {
      "type" : "text",
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "ignore_above" : 256
        }
      }

I created (using Kibana) a scripted field named "the_sf"

And I tried this painless code:

Map m = new HashMap();
m.put("Yes", 1);
m.put("No", 2);
m.put("Maybe", 3);
m.put("British Columbia", 10);
m.put("Alberta", 20);
here_ONE_of_the_commands_bellow

return m.get(doc['province.keyword'].value)

Preview results = 
[
 {
  "_id": "ct_jUG0BloL_9a5G0aRL",
  "the_sf": [
   10         ------------------------ >>>  YEH !!!!
  ]
 },
(...)

return m.get(doc['province.keyword'])

Preview results = 
[
 {
  "_id": "ct_jUG0BloL_9a5G0aRL",
  "the_sf": [
   null    ------------------- >>> Makes sense, since doc['field'] is a pointer and not the value...
  ]
 },
(...)

return m.get(doc['province'].value)

Preview results = 
{
 "root_cause": [
  {
   "type": "script_exception",
   "reason": "runtime error",
   "script_stack": [
  "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:759)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:290)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:101)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:98)",
"java.base/java.security.AccessController.doPrivileged(AccessController.java:310)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:98)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"return m.get(doc['province'].value)",   ------------------- >>> WHY ????
"                 ^---- HERE"
   ],
   "script": "Map m = new HashMap();\n\nm.put(\"Not yet\", 1);\nm.put('Less than once a month', 2);\nm.put('A few times a month', 3);\nm.put('About once a week', 4);\nm.put('A few times a week', 5);\nm.put('Most days or every day', 6);\nm.put(\"British Columbia\", 10);\n\nreturn m.get(doc['province'].value)",
   "lang": "painless"
  }
 ],
(...)

return m.get(doc['province'])

Preview results =
The same as above, but
"return m.get(doc['province'])",  ------------------- >>> WHY ??? Well, OK it is a pointer and not the value, but it should return null as before, don't?
"                 ^---- HERE"

The issue is that I tested also with the "q_1" field and I got different results/errors

return m.get(doc['q_1.keyword'])

[
 {
  "_id": "ct_jUG0BloL_9a5G0aRL",
  "the_sf": [
   null    ------------------- >>> Makes sense, since doc['field'] is a pointer and not the value...
 ]
}, (...)

return m.get(doc['q_1.keyword'].value)

{
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:496)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:503)",
"return m.get(doc['q_1.keyword'].value)", ------------------- >>> WHY ??? I know this message is big, but if you remember up there, it worked with doc['province.keyword'].value
" ^---- HERE"
], (...)

return m.get(doc['q_1'])
return m.get(doc['q_1'].value)

Same errors as doc['province'] and doc['province'].value

Sorry for the huge message... I hope someone will be patient enough to read everything and answered it.

Cheers,
Paulo

[abdon (Abdon Pijpelink)]

There are a few things going on here.

When you access a field using the doc['field'] notation you are accessing so called "doc values". Doc values are one of the many types of data structures that Lucene creates out of your documents when you index those. The thing with doc values for string fields is that they are only available for keyword fields and not for text fields.

This is why the following works:

return m.get(doc['province.keyword'].value)

but this fails:

return m.get(doc['province'].value)

Now, why does return m.get(doc['q_1.keyword'].value) fail? It's hard to say, but there should be a reason section in the error that Elasticsearch returns that explains it. I suspect that you have at least one document that does not have a q_1 field. Try checking whether a document actually contains q_1 values:

if (doc['q_1.keyword'].size() != 0) {
   return m.get(doc['q_1.keyword'].value)
}

[pmarques (Paulo Marques)]

Hi @abdon

Thank you so much for your answer. It is true that not every doc has the q_1 field.

I tried this before.

if (!doc['q_1.keyword'].empty) {
   return m.get(doc['q_1.keyword'].value)
}

But it didn't work.

Thanks for the .size() tip. It worked well.

Cheers,
Paulo