Enrich Policy Execute Unable to Find Source Index

cj_hillbrand · April 20, 2022, 1:04am

Hello,
I am trying to familiarize myself with the Enrich Policy. I have set up a source index and an enrich policy. When I go to execute the enrich policy I receive a "index_not_found_exception".

I have posted some short outputs of the index I'm trying to attach the policy to. Any help or guidance is more than welcome. (as an FYI I have been following this tutorial.

On an unrelated note, is it possible to create an enrichment policy based off of an alias?

I know output should be linked, but hopefully this isn't too much to warrant any indirection.

GET _indices
yellow open node_attributes-46121119122022   dJ2MaS-3QXqp-CK5lhYrbg 1 1 2959807 0     1gb     1gb

GET _enrich/policy
{
        "match" : {
          "name" : "node_enrich_policy_2",
          "indices" : [
            "node_attributes-46121119122022"
          ],
          "match_field" : "nodeId",
          "enrich_fields" : [ ... ]
    }
}

cj_hillbrand · April 20, 2022, 1:06am

I should have posted the output of the error as well:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [.tasks]",
        "resource.type" : "index_expression",
        "resource.id" : ".tasks",
        "index_uuid" : "_na_",
        "index" : ".tasks"
      }
    ],
    "type" : "resource_not_found_exception",
    "reason" : "task [qz0ey6USQa2Zq5kgeB-77w:72701] isn't running and hasn't stored its results",
    "caused_by" : {
      "type" : "index_not_found_exception",
      "reason" : "no such index [.tasks]",
      "resource.type" : "index_expression",
      "resource.id" : ".tasks",
      "index_uuid" : "_na_",
      "index" : ".tasks"
    }
  },
  "status" : 404
}```

stephenb · April 20, 2022, 1:22am

Hi @cj_hillbrand Welcome to the community

Can you show the exact enrich execute command you ran?

What version of the stack are you running?

Did you happen to delete any system Indices? Indices that star with a .

cj_hillbrand · April 20, 2022, 1:25am

Hey Stephen,

Firstly I ran this to create the enrich policy:

PUT /_enrich/policy/node_enrich_policy_2
{
  "match": {
    "indices": "node_attributes-46121119122022",
    "match_field": "nodeId",
    "enrich_fields": [
      "rackLocation",
      "machinePool",
      "hardwareSku",
      "bios",
      "os",
      "bmc",
      "microcode",
      "cpuId",
      "cpuDescription",
      "ssdModels",
      "ssdFirmware",
      "socFirmware",
      "nitroFirmware",
      "cerberusVersion",
      "region",
      "isUtilitySku",
      "isStorageSku"
      ]
  }
}

and my knowledge is that the second command does not have much variability, but for completeness:

PUT /_enrich/policy/node_enrich_policy_2/_execute?wait_for_completion=false

Thank you for the help.

cj_hillbrand · April 20, 2022, 1:27am

Had missed the latter two asks:
I am running version 8.1.2 and I do not recall intentionally removing any system indices.

stephenb · April 20, 2022, 1:27am

Try Take off the wait_for_completion

What user are you logged in as?

I asked because the execute the enrich policy requires

The enrich_user built-in role

cj_hillbrand · April 20, 2022, 1:35am

Taking off the wait for completion results in a request timeout. Although tracking the task, I find that it results in the same error message as the initial post.

As this is a test node (just running on my local machine), I have disabled all authentication/authorization. I would imagine even with authN/authZ disabled I still have some execution policy enabled with its associated roles.

I tried checking with GET _security/role but am receiving: Incorrect HTTP method for uri what else would you suggest to check if the enrich_user role is assigned to my current user?

stephenb · April 20, 2022, 1:58am

Hmmm.. I Have not turned all security off as 8.x is supposed to be secured by default.

And system indices etc are more protected etc.

So you are not logged in at all?....all security is turned off? (Exactly what steps / settings?)... that seems harder than just setting up the automatic security ... but I understand.

The error indicates the user / role does not have permission to interact with the tasks, and the .tasks system index ..that I suspect is the issues... the default user may not have those permissions or something.

With security completely off I do not think you can access/update change roles etc.

I just ran through and end-to-end sample enrichment index create, execute etc on my default secured and it worked 8.1.2.

I would need to set up and unsecured 8.1.2 node, but I suspect that is your issue.

What is the output of

GET _cat/indices/.*/?v&s=index

cj_hillbrand · April 20, 2022, 2:01am

Here is the portion of the .yml that I had change to fulfill my interpretation of turning off security:

# Enable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

Here is the output from the command you had requested:

health status index                                                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .apm-agent-configuration                                      CfLfAxs7TXu3XTfkwrTnNA   1   0          0            0       225b           225b
green  open   .apm-custom-link                                              kdKLd4GeSXei8v9h8R3ypg   1   0          0            0       225b           225b
green  open   .ds-.logs-deprecation.elasticsearch-default-2022.04.19-000001 hrWClk9TRhyCtZf-xNifPQ   1   0          1            0     11.2kb         11.2kb
green  open   .ds-ilm-history-5-2022.04.19-000001                           zVvkd0CeQpCSfng9b2RSFQ   1   0          9            0     28.6kb         28.6kb
green  open   .enrich-node_enrich_policy-1650413707748                      epTrQE5SSnyvnxEygy35hA   1   0    2959807            0    537.8mb        537.8mb
green  open   .enrich-node_enrich_policy_1-1650415618467                    6Ed83434TDSHhbDKOycYqA   1   0    2959807            0    537.7mb        537.7mb
green  open   .enrich-node_enrich_policy_2-1650418071115                    cTpaGsTLS_KrgV04Nm35ZQ   1   0    2959807            0    537.8mb        537.8mb
green  open   .geoip_databases                                              x6SkxBMbSUeyn9N5D_T-4A   1   0         40            0     37.9mb         37.9mb
green  open   .kibana-event-log-8.1.2-000001                                3tOHOtQwRVGRuIOIoDnINA   1   0          1            0      6.2kb          6.2kb
green  open   .kibana_8.1.2_001                                             ccbdxoLuSnmVPTzhwrQhVQ   1   0         60            0      4.7mb          4.7mb
green  open   .kibana_task_manager_8.1.2_001                                6gxCWPVYQtGH5yxkUyrQ1A   1   0         19         7534        1mb            1mb
green  open   .security-7                                                   cO6Ez5wWQSWIg0sWYze2Yw   1   0          2            0       11kb           11kb

stephenb · April 20, 2022, 2:06am

You are missing the .tasks index... why I am not sure
green open .tasks 7X4dvKi7RSS2b-pKzJELmQ 1 0 12 0 23.6kb 23.6kb

perhaps restart the node...

What is the output of

GET /_tasks

If I get a chance I will try an unsecured node...

OHHH it looks like you created some actual enrich policies...

green open .enrich-node_enrich_policy_2-1650418071115 cTpaGsTLS_KrgV04Nm35ZQ 1 0 2959807 0 537.8mb 537.8mb

Looks like you are having task management issues... (which is not good)

Did you try to actually run an ingest pipeline with the enrich?

cj_hillbrand · April 20, 2022, 2:09am

I have pasted the output of GET /_tasks below. For the sake of my learning what is the role of the .tasks index?
I will also go ahead and try and restart the instance and see if anything changes.

{
  "nodes" : {
    "qz0ey6USQa2Zq5kgeB-77w" : {
      "name" : "LAPTOP-QAJMR977",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "attributes" : {
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "8455716864",
        "ml.machine_memory" : "16907538432"
      },
      "tasks" : {
        "qz0ey6USQa2Zq5kgeB-77w:132294" : {
          "node" : "qz0ey6USQa2Zq5kgeB-77w",
          "id" : 132294,
          "type" : "direct",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1650420442108,
          "running_time_in_nanos" : 90600,
          "cancellable" : false,
          "parent_task_id" : "qz0ey6USQa2Zq5kgeB-77w:132293",
          "headers" : {
            "trace.id" : "9592fb1f2914efb59f85de50c2183e9e"
          }
        },
        "qz0ey6USQa2Zq5kgeB-77w:132293" : {
          "node" : "qz0ey6USQa2Zq5kgeB-77w",
          "id" : 132293,
          "type" : "transport",
          "action" : "cluster:monitor/tasks/lists",
          "start_time_in_millis" : 1650420442108,
          "running_time_in_nanos" : 253600,
          "cancellable" : false,
          "headers" : {
            "trace.id" : "9592fb1f2914efb59f85de50c2183e9e"
          }
        },
        "qz0ey6USQa2Zq5kgeB-77w:19" : {
          "node" : "qz0ey6USQa2Zq5kgeB-77w",
          "id" : 19,
          "type" : "persistent",
          "action" : "geoip-downloader[c]",
          "start_time_in_millis" : 1650409116921,
          "running_time_in_nanos" : 11325187288501,
          "cancellable" : true,
          "cancelled" : false,
          "parent_task_id" : "cluster:2",
          "headers" : { }
        }
      }
    }
  }
}

stephenb · April 20, 2022, 2:11am

AFAIK it keeps track of the actual tasks... state etc.

cj_hillbrand · April 20, 2022, 2:21am

I went ahead and created an ingestion pipeline, and threw some documents through it and it is working as expected! Interesting the error message had popped up initially and I am missing the .tasks index.

I ran GET _enrich/_stats to evaluate if my policy had been executed. Do you have any suggestions on what I can do to make sure the policy is executed correctly? (aside from attempting to use it in an ingestion pipeline)

Thank you for all your help thus far.

stephenb · April 20, 2022, 2:26am

Ok I just set up an Elasticsearch and Kibana absolutely no security. (Thanks for prompting me to do that)

I can not see the .tasks index as expected BUT

I just did and end to end enrich and it worked fine...

Somehow your node / user / data or something is in a bit of a funky state,

I would restart the node

If this is a test machine I would try clearing and setup from scratch ...

You can look at the size / count of docs in the enrich policy... it is just an optimized index the number of docs should match.

GET _cat/indices/.enrich*/?v&s=index

health status index                                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .enrich-asset-download-policy-1650421161696 j-QvDfYTSB6PyHn470k0xQ   1   0          3            0      3.7kb          3.7kb

Sorry I am not sure what your tasks issue is...

cj_hillbrand · April 20, 2022, 6:08pm

Thanks for doing all that work on your end in attempt to replicate the issue! Since it is just a test machine, ill go ahead and wipe clean and start over, but I think I have enough information to debug if something like this arises again.

So Ill mark your last response as a solution, so we can have this thread resolve itself.

system · May 18, 2022, 6:09pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.