Access a field's value in elastic search without indexing

[Harinder_Singh]

Hi, I have a requirement to form a script query from java code where it has to formulate the final score after getting the elastic score plus referring a variable on the runtime.

document is indexed like below

{
  "ranking": {
    "names": [
      "john",
      "smith",
      "paul"
    ],
    "grades": {
      "john": "12",
      "smith": "90",
      "paul": "18"
    }
  }
}

There will be many documents like below. I am not indexing the grades field as it will have various numbers of users and can explode the mapping limits. On runtime I want the score to be combination as below

{
  "script_score": {
    "query": {
      "bool": {
        "must": [
          {
            "match_phrase": {
              "ranking,names": {
                "query": "john"
              }
            }
          }
        ]
      }
    },
    "script": {
      "source": "def score = 0.0; try{ score = doc['ranking.grades.john'].value + _score; }catch(Exception exception){ score = _score;} return score;"
    }
  }
}

I have tried below ways

1. Without indexing , I can get it through _source , but _source lookup is very slow and it is resulting in perfromance bottlenecks

2. have tried through doc and field lookup, they are fast , but they need ranking.grades fields to be indexed. Down side of this is as the names cannot have any limit, as soon as some 1000 names (mapping default limit) are indexed , the next insertion starts failing.

I am looking for some way where I can access the value without getting it indexed but not at the cost of high latency.

[leandrojmp (Leandro Pereira)]

@Harinder_Singh

I am not indexing the grades field as it will have various numbers of users and can explode the mapping limits.

What does your mapping looks like? You may map the grades field as flattened, this will avoid the mapping explosion.

From the documentation:

This data type can be useful for indexing objects with a large or unknown number of unique keys. Only one field mapping is created for the whole JSON object, which can help prevent a mappings explosion from having too many distinct field mappings.

This seems to be your case with the grades field.

[Harinder_Singh]

@Harinder_Singh

{
  "ranking": {
    "names": [
      "john",
      "smith",
      "paul"
    ],
    "grades": {
      "john": "0.3412",
      "smith": "0.23490",
      "paul": "0.23418"
    }
  }
}

Hi @leandrojmp ,

Keeping type as flattened resolves the mapping issue. But in script I have to make the casting Double.parseDouble to get the value.

In the actual index, the values under grades will be in decimal.

Mapping is as below

"mappings": {
    "properties": {
      "rankings": {
        "properties": {
          "names": {
            "type": "keyword"
          },
          "grades": {
            "type": "flattened"
          }
        }
      }
    }
  }

Query is as below

GET /performance/_search
{
  "query": {
    "bool":{
      "must": [
        {
          "bool": {
            "minimum_should_match": "1",
            "should": [
              {
                "script_score": {
                  "query": {
                    "bool": {
                      "must": [
                        {
                          "match_phrase": {
                            "ranking.names": {
                              "query": "john"
                            }
                          }
                        }
                      ]
                    }
                  },
                  "script": {
                    "source": "return Double.parseDouble(doc['ranking.grades.john'].value) + _score;"
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}

[leandrojmp (Leandro Pereira)]

@Harinder_Singh

Keeping type as flattened resolves the mapping issue. But in script I have to make the casting Double.parseDouble to get the value.

In the actual index, the values under grades will be in decimal.

Not sure if I got the issue, but every child in the flattened field will be mapped as keyword, so if you want to work with them as numbers you will probably need to convert it in your script.