Nested data not shown correctly with painless script

[junkoshimane]

Hi all,

I may find issue with the combination of "nested data" and "painless script".

• In case "painless script" is NOT used, documents are shown correctly.
• In case "painless script" is used, they are shown wrongly. Of course, I never update documents in the script.

Please help to solve issue or analyze it further.

---

• mapping

{
  "mappings": {
    "_doc": {
      "properties": {
        "study_id": {
          "type": "keyword"
        },
        "ext_number":{
          "type": "keyword"
        },
        "study_timestamp":{
          "type": "date"
        },
        "study_secondslong":{
          "type": "float"
        },
        "student_learn_rate":{
          "type": "float"
        },
        "additional_study_timestamp":{
          "type": "date"
        },
        "additional_data01":{
          "type": "keyword"
        },
        "additional_data02":{
          "type": "keyword"
        },
        "study_text":{
          "type": "text",
          "analyzer" : "kuromoji"
        },
        "study_text_student":{
          "type": "text",
          "analyzer" : "kuromoji"
        },
        "study_text_teacher":{
          "type": "text",
          "analyzer" : "kuromoji"
        },
        "file_uri":{
          "type": "keyword"
        },
        "book_del_flg":{
          "type": "keyword"
        },
        "teach_cm":{
          "type": "double"
        },
        "student_cm":{
          "type": "double"
        },
        "claim_old":{
          "type": "integer"
        },
        "negative_task":{
          "type": "text"
        },
        "negative_score":{
          "type": "float"
        },
        "teacher":{
          "type": "keyword"
        },
        "teacher_flg":{
          "type": "keyword"
        },
        "claim":{
          "type": "integer"
        },
        "nested_data":{
          "type": "nested",
          "properties":{
            "learner_id":{
              "type": "keyword"
            },
            "version_number":{
              "type": "float"
            },
            "dict_name":{
              "type": "keyword"
            },
            "c_time":{
              "type": "float"
            },
            "u_time":{
              "type": "float"
            },
            "m_time1":{
              "type": "float"
            },
            "m_time2":{
              "type": "float"
            },
            "m_time3":{
              "type": "float"
            },
            "m_time4":{
              "type": "float"
            },
            "tf_done_flg":{
              "type": "keyword"
            },
           "cluster_done_flg":{
              "type": "keyword"
            },
            "evalexp_done_flg":{
              "type": "keyword"
            },
            "learner_data":{
              "type": "nested",
              "properties":{
                "study_id": {
                  "type": "keyword"
                },
                "turn":{
                  "type": "integer"
                },
                "learn_begin_timeoffset":{
                  "type": "float"
                },
                "learn_end_timeoffset":{
                  "type": "float"
                },
                "learn_confidence":{
                  "type": "keyword"
                },
                "words_confidence":{
                  "type": "text"
                },
                "get_words":{
                  "type": "text",
                  "fielddata": true
                },
                "present_rate":{
                  "type": "float"
                },
                "silent_time":{
                  "type": "float"
                },
                "learn_position":{
                  "type": "integer"
                },
                "study_reason_flg":{
                  "type": "keyword"
                },
                "_words_confidence":{
                  "type": "text"
                }
              }
            }
          }
        }
      }
    }
  }
}

• only mutch_all query. NOP

GET school/_search
{
  "query": {
    "match_all": {}
  }
}

• When returning "params._source" from painless script, the nested data is wrong for some documents.

GET school/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields": {
    "test": {
      "script": {
        "lang": "painless",
        "source": "
          return params._source
        "
      }
    }
  }
}

• However, when returning "params._source.nested_data" from painless script, the nested data is correct for all documents.

GET school/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields": {
    "test": {
      "script": {
        "lang": "painless",
        "source": "
          return params._source.nested_data
        "
      }
    }
  }
}

[junkoshimane]

None???

[junkoshimane]

None??? again

[spinscale (Alexander Reelsen)]

Can you please come up with a fully fledged example include samples documents? It is really hard to understand or even interpret what you mean with wrong and correct, if you do not include responses and sample documents.

Thanks!

--Alex

[junkoshimane]

Hi Alex,

Thank you for your response.
I found the issue with documents including Japanese language but have not reproduced it in English yet.
So I have not been able to post sample documents to this topic.

After I reproduce the issue in English, I will let you know.

By the way, is there any way (e.g. log with debug level) to analyze this issue further by myself?
As far as I looked for such way, I could't.

[junkoshimane]

• FYI

ElasticSearch Painless script: How to iterate in an array of Nested Objects

Painless: problem accessing nested object using dot notation

Hi, I'm having problems reindexing data using a painless script. The nested field context.data.result is mapped as an object in the destination index, but in some cases the source data is either a string or an array of longs The script attempts to: Check if the field exists Check the field's type If it is a string or array create a temporary string / array variable assign the value of the offending field to this variable change the value of the field to be a new object (passed in via param…

Painless access to nested objects

Hi, I'm facing the following situation : My data has a structure similar to this { "site1" : { "events": [ {"event1", "date1"}, {"event1","date1-2"},{"event2","date3"}}] ... }, "site2" : { "events": [ {"event2", "date4"}, {"event1","date5"},{"event3","date6}}] ... }, ... } What I want is : Count the number of sites where a specific event occurs during a given time range. Count for each site the numer of occurences during a specific time range. I could get the first via a simple …

Get nested fields data with Painless scripting

I'm trying to access nested fields data through Painless inline scripting and can't seem to be able to get it to work. Without getting into too many details, the reason for this is that we have certain fields that support a default value, and an optional custom override. To avoid having to return both fields and then have the application decide which one to use, I'd like to use a scripted field to check if the override has value and return that, or otherwise fallback to the default value. I've …

Use normal and nested fields in same Painless script query?

I have a mapping like this: { printings: { type: "nested", properties: { prop1: {type: "number"} } }, prop2: {type: "number"} } I then want to build a Painless query like this: "script": { "lang": "painless", "inline": "doc['prop1'] > (3 * doc['printings.prop2'])" } However testing this in Sense doesn't work. If I replace the nested prop2 with a simple number then it does work. Is there a way to access both root and nested props in a sin…