Scripted field from multiple indices

Elastic Stack Elasticsearch
1

I want to create a script field that uses information in another index to create a field.
suppose my first index is index1 which reads packets from interface traffic and includes fields "ip-address" and "mac-address".
I have another index, index2 which includes a field named "IP" which keeps ip addresses and a field named "MAC" which keeps mac addresses and a field named as "machine-FQDN" that keeps machine names (basically this index is in fact a small table that has stored information about machines in my network)

Then I want for each packet to create a script field in index1 which reads "machine-FQDN" from index2 based on the "ip-address" and "mac-address" fields of index1.

Here is how I think I should do it:

GET /_search
{
    "_source": {
      "includes": ["machine-FQDN"]  },
    "query" : {
      "script_fields" : {
        "script" : {
          "lang": "painless",
          "bool" :{
            "must":[
             "IP":"doc['ip-address'].value",
              "MAC":"doc['mac-address'].value"
              ]
          }
        }    
      }
    }
}

but when I run it in Dev Tools I get:

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "no [query] registered for [script_fields]",
        "line": 5,
        "col": 25
      }
    ],
    "type": "parsing_exception",
    "reason": "no [query] registered for [script_fields]",
    "line": 5,
    "col": 25
  },
  "status": 400
}

Even when I do it as:

GET /_search
{
    "_source": {
      "includes": ["machine-FQDN"]  },
    "query" : {
      "script_fields" : {
        "script" : {
          "lang": "painless",
          "bool" :{
            "must":[
             {"match":{"query":{"IP":"doc['ip-address'].value"}}},
              {"match":{"query":{"MAC":"doc['mac-address'].value"}}}
              ]
          }
            
      }
    }
    }
}

I receive exactly the same error.

When I run the above code in Management in scripted fields I receive following error:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["GET /_search\n{\n    \"_source\": {\n       ...","             ^---- HERE"],"script":"GET /_search\n{\n    \"_source\": {\n      \"includes\": [\"machine-FQDN\"]  },\n    \"query\" : {\n      \"script_fields\" : {\n        \"script\" : {\n          \"lang\": \"painless\",\n          \"bool\" :{\n            \"must\":[\n             {\"match\":{\"query\":{\"IP"\":\"doc['ip-address'].value\"}}},\n              {\"match\":{\"query\":{\"POP-id\":\"doc['mac-address'].value\"}}}\n              ]\n          }\n            \n      }\n    }\n    }\n}","lang":"painless"},{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting","bytes_wanted":0,"bytes_limit":0}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"nprobe-2017.05.18","node":"Lzhq9rYeTpmDIwWygXrS3w","reason":{"type":"script_exception","reason":"compile error","script_stack":["GET /_search\n{\n    \"_source\": {\n       ...","             ^---- HERE"],"script":"GET /_search\n{\n    \"_source\": {\n      \"includes\": [\"machine-FQDN\"]  },\n    \"query\" : {\n      \"script_fields\" : {\n        \"script\" : {\n          \"lang\": \"painless\",\n          \"bool\" :{\n            \"must\":[\n             {\"match\":{\"query\":{\"IP"\":\"doc['ip-address'].value\"}}},\n              {\"match\":{\"query\":{\"MAC\":\"doc['mac-address'].value\"}}}\n              ]\n          }\n            \n      }\n    }\n    }\n}","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"unexpected token ['{'] was expecting one of [{<EOF>, ';'}]."}}},{"shard":1,"index":"nprobe-2017.05.21","node":"Lzhq9rYeTpmDIwWygXrS3w","reason":{"type":"general_script_exception","reason":"Failed to compile inline script [GET /_search\n{\n    \"_source\": {\n      \"includes\": [\"machine-FQDN\"]  },\n    \"query\" : {\n      \"script_fields\" : {\n        \"script\" : {\n          \"lang\": \"painless\",\n          \"bool\" :{\n            \"must\":[\n             {\"match\":{\"query\":{\"IP"\":\"doc['ip-address'].value\"}}},\n              {\"match\":{\"query\":{\"MAC\":\"doc['mac-address'].value\"}}}\n              ]\n          }\n            \n      }\n    }\n    }\n}] using lang [painless]","caused_by":{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting","bytes_wanted":0,"bytes_limit":0}}}]},"status":500}
    at http://10.0.2.11:5601/bundles/kibana.bundle.js?v=15063:230:1333
    at Function.Promise.try (http://10.0.2.11:5601/bundles/commons.bundle.js?v=15063:98:28179)
    at http://10.0.2.11:5601/bundles/commons.bundle.js?v=15063:98:27549
    at Array.map (native)
    at Function.Promise.map (http://10.0.2.11:5601/bundles/commons.bundle.js?v=15063:98:27504)
    at callResponseHandlers (http://10.0.2.11:5601/bundles/kibana.bundle.js?v=15063:230:949)
    at http://10.0.2.11:5601/bundles/kibana.bundle.js?v=15063:229:20482
    at processQueue (http://10.0.2.11:5601/bundles/commons.bundle.js?v=15063:38:23621)
    at http://10.0.2.11:5601/bundles/commons.bundle.js?v=15063:38:23888
    at Scope.$eval (http://10.0.2.11:5601/bundles/commons.bundle.js?v=15063:39:4619)

I appreciate if you could help me with this.

2

You cannot do this unfortunately.

3

Hi Mark,

Thanks for the reply. Do you have any idea to implement this in another way, or you believe it is not possible at all?
I thought maybe I can use script processor to implement this. Do you think it would be possible to do it there?

4

There is no such thing as a "script_fields" query. That is exactly what the error message is telling you.

This sounds like something you should do from your application. It is essentially a join (you want to get your result documents, then run another search and use script_fields with that query).

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.