What is wrong with my JSON Input usage?


(Nestor Manrique) #1

Hello,

I have an index where I store some server information I pull from an http API, I'm trying to get a pie chart with the number of vcpus each of the server has available. I have to multiply the 'totalProcessorCores' by 2 to get the vcpus (threads) but for some reason I'm getting either of this messages depending of the request I'm sending to elasticsearch.

Visualize: [parsing_exception] Unknown key for a START_OBJECT in [script]

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "analyze_wildcard": true,
            "query": "type: Server"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1491195600000,
              "lte": 1491281999999,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "terms": {
        "field": "id.name",
        "size": 20,
        "order": {
          "_term": "desc"
        }
      },
      "aggs": {
        "1": {
          "top_hits": {
            "docvalue_fields": [
              "totalProcessorCores"
            ],
            "_source": "totalProcessorCores",
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ],
            "script": {
              "lang": "groovy",
              "inline": "doc['totalProcessorCores'].value * 2"
            }
          }
        }
      }
    }
  }
}

Visualize: [parsing_exception] Unknown key for a VALUE_STRING in [script]

{
  .
  .
  .,
            "script": "doc['totalProcessorCores'].value * 2",
            "lang": "groovy"
          }
        }
      }
    }
  }
}

I also allowed the groovy scripts in my elasticsearch.yml and restarted the service.

script.engine.groovy.inline.aggs: true
script.engine.groovy.inline.search: true

In some posts I found googling this errors they were always related to some typo or bad use of the DSL but I don't seem to catch mine. I already checked it agains the current docs and also tried this example with no success.

Thank you,

N


(Jimmy Kuang) #2

Can you try the following?

GET YOUR_INDEX/YOUR_DOC/_search
{
  "script_fields": {
    "my_doubled_field": {
      "script": {
        "lang":   "expression",
        "inline": "doc['totalProcessorCores'] * multiplier",
        "params": {
          "multiplier": 2
        }
      }
    }
  }
} 

(Nestor Manrique) #3

Hello jkuang,

Tried. Same "Unknown key for a START_OBJECT in [script]." error shows up.

N


(Jimmy Kuang) #4

I think it's the way the doc['totalProcessorCores'] field object is being reference. The code says it can't match this object.

Can you provide your index mapping and a line of json input?


(Nestor Manrique) #5

Sure, this is the mapping for the server type.

    "mappings" : {
      "Server" : {
        "dynamic" : false,
        "properties" : {
          "biosVersion" : {
            "type" : "keyword"
          },
          "coresPerProcessorSocket" : {
            "type" : "integer"
          },
          "serverPoolId" : {
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "memory" : {
            "type" : "long"
          },
          "processorType" : {
            "type" : "keyword"
          },
          "biosReleaseDate" : {
            "type" : "keyword"
          },
          "ethernetPortIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "threadsPerCore" : {
            "type" : "integer"
          },
          "serverRunState" : {
            "type" : "keyword"
          },
          "clusterId" : {
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "processorSpeed" : {
            "type" : "double"
          },
          "productName" : {
            "type" : "keyword"
          },
          "manufacturer" : {
            "type" : "keyword"
          },
          "biosVendor" : {
            "type" : "keyword"
          },
          "hostname" : {
            "type" : "keyword"
          },
          "protected" : {
            "type" : "boolean"
          },
          "networkIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "enabledProcessorCores" : {
            "type" : "integer"
          },
          "managerUuid" : {
            "type" : "keyword"
          },
          "storageInitiatorIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "ovmVersion" : {
            "type" : "keyword"
          },
          "serialNumber" : {
            "type" : "keyword"
          },
          "repositoryExportIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "totalProcessorCores" : {
            "type" : "integer"
          },
          "populatedProcessorSockets" : {
            "type" : "integer"
          },
          "vmIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "usableMemory" : {
            "type" : "long"
          },
          "storageElementIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "maintenanceMode" : {
            "type" : "boolean"
          }
        }
      },

About the line of json input well I just simply want to multiply that value by 2, or if it's possible by the threadsPerCore value which is another integer in the same type mapping, so the script I wrote in the OP stands as an example.

Thank you very much!


(Jimmy Kuang) #6

I think something is wrong with your index. I couldn't reproduce your issue. Please see my sample below using your mapping:

PUT mymulti
{
 "mappings" : {
      "Server" : {
        "dynamic" : false,
        "properties" : {
          "biosVersion" : {
            "type" : "keyword"
          },
          "coresPerProcessorSocket" : {
            "type" : "integer"
          },
          "serverPoolId" : {
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "memory" : {
            "type" : "long"
          },
          "processorType" : {
            "type" : "keyword"
          },
          "biosReleaseDate" : {
            "type" : "keyword"
          },
          "ethernetPortIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "threadsPerCore" : {
            "type" : "integer"
          },
          "serverRunState" : {
            "type" : "keyword"
          },
          "clusterId" : {
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "processorSpeed" : {
            "type" : "double"
          },
          "productName" : {
            "type" : "keyword"
          },
          "manufacturer" : {
            "type" : "keyword"
          },
          "biosVendor" : {
            "type" : "keyword"
          },
          "hostname" : {
            "type" : "keyword"
          },
          "protected" : {
            "type" : "boolean"
          },
          "networkIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "enabledProcessorCores" : {
            "type" : "integer"
          },
          "managerUuid" : {
            "type" : "keyword"
          },
          "storageInitiatorIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "ovmVersion" : {
            "type" : "keyword"
          },
          "serialNumber" : {
            "type" : "keyword"
          },
          "repositoryExportIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "totalProcessorCores" : {
            "type" : "integer"
          },
          "populatedProcessorSockets" : {
            "type" : "integer"
          },
          "vmIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "usableMemory" : {
            "type" : "long"
          },
          "storageElementIds" : {
            "type" : "nested",
            "properties" : {
              "name" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "value" : {
                "type" : "keyword"
              },
              "uri" : {
                "type" : "keyword"
              }
            }
          },
          "maintenanceMode" : {
            "type" : "boolean"
          }
        }
      }
 }}
 
 POST mymulti/Server/
 {
   "totalProcessorCores" : 2
 }
 GET mymulti/Server/_search
{
  "script_fields": {
    "my_doubled_field": {
      "script": {
        "lang":   "expression",
        "inline": "doc['totalProcessorCores'] * multiplier",
        "params": {
          "multiplier": 2
        }
      }
    }
  }
}

(Jimmy Kuang) #7

Results:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "mymulti",
        "_type": "Server",
        "_id": "AVs6uNoNnn3k1Vj753OV",
        "_score": 1,
        "fields": {
          "my_doubled_field": [
            4
          ]
        }
      },
      {
        "_index": "mymulti",
        "_type": "Server",
        "_id": "AVs6vWEjnn3k1Vj753OW",
        "_score": 1,
        "fields": {
          "my_doubled_field": [
            6
          ]
        }
      }
    ]
  }
}

(Nestor Manrique) #8

I tried your code again in the dev tools tab in kibana and it shows the same results. However I still get the errors if I try to do the same in a visualization json input. For instance,

And in the input json I wrote:

{
"script": {
        "lang":   "expression",
        "inline": "doc['totalProcessorCores'] * multiplier",
        "params": {
          "multiplier": 2
        }
      }
}

But still got the Unknown key for a START_OBJECT in [script] error.


(Jimmy Kuang) #9

What version of ES and Kibana are you using?


(Nestor Manrique) #10

I'm using 5.3 for everything.


(Jimmy Kuang) #11

I'm using 5.3 too and my sample works, but it doesn't work for you. Just to confirm, you're getting the error in dev tools or in the visualization?


(Nestor Manrique) #12

In the visualization. Here is a bigger print screen.

I couldn't fit the json input field in the image but you can read what I wrote 2 comments ago.


(Jimmy Kuang) #13

I'm able to reproduce your error now. This is a problem with the JSON input for sure. I even used the example from the doc and I am getting your error.

https://www.elastic.co/guide/en/kibana/current/metric-chart.html

There is a workaround. You can go to Kibana > Management > Index Pattern > choose your index > click on the scripted field tab and put the script there. Then go back to visualization and choose the scripted field .


(Nestor Manrique) #14

Hello jkuang,

Thank you very much for your help. Indeed making a new scripted field worked like a charm, do you consider a good idea to open an issue in the github page? Did you already do it by any chance?

Thank you very much again!

N


(Jimmy Kuang) #15

I would recommend you open an issue on github. ^^ I'm sure many people would encounter this too. Helping each other is what open source is about. :smiley:


(Nestor Manrique) #16

All right, I just opened an issue to address this matter.

Thank you again jimmy!

N


(system) closed #17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.