# Kafka Integration Installation Error - "Limit of total dimension fields \[16\] has been exceeded"

**URL:** <https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052>\
**Category:** Elastic Agent\
**Tags:** fleet\
**Created:** [July 24, 2023, 8:21am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052 "2023-07-24T08:21:47Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Moaath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moaath/32/121792_2.png) [@Moaath](https://discuss.elastic.co/u/Moaath)\
**Post date:** [July 24, 2023, 8:21am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/1 "2023-07-24T08:21:47Z")

</div>

Hi Everyone,

I am currently trying to install the Kafka integration via the Fleet UI in Kibana, but I've encountered an issue that I hope someone can shed some light on.

The error message I'm receiving is as follows:

```auto
Error installing kafka 1.7.0: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded.

```

In an attempt to resolve this issue, I checked the index.mapping.dimension\_fields.limit setting and found that the limit is indeed set to 16, as highlighted in the error message. However, I was unable to find a way to update or increase this limit.

Elastic Stack Version: 8.5.4

Regards,

---

<div class="post-metadata">

**Author:** ![coezdemir](https://avatars.discourse-cdn.com/v4/letter/c/7c8e57/32.png) [@coezdemir](https://discuss.elastic.co/u/coezdemir)\
**Post date:** [July 24, 2023, 9:51am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/2 "2023-07-24T09:51:59Z")

</div>

Hello,  
Never used Kafka integration before, [but you can update this setting on your index.](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html)

just follow the examples. Keep in mind that elasticsearch have defaults for settings like these for a reason, increasing it might reduce the performance.

Regards.

---

<div class="post-metadata">

**Author:** ![Julia\_Bardi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julia_bardi/32/79463_2.png) [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)\
**Post date:** [July 24, 2023, 12:26pm UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/3 "2023-07-24T12:26:47Z")

</div>

Thanks coezdemir for the help, here is the correct doc link: [Time series index settings | Elasticsearch Guide [8.5] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/tsds-index-settings.html)

---

<div class="post-metadata">

**Author:** ![Moaath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moaath/32/121792_2.png) [@Moaath](https://discuss.elastic.co/u/Moaath)\
**Post date:** [July 25, 2023, 8:34am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/4 "2023-07-25T08:34:30Z")

</div>

Thank you all.

I have updated the 'index.mapping.dimension\_fields.limit' in the '.fleet\_globals-1' component template, since all fleet-managed index templates inherit settings from this template. However, despite these changes, I am still encountering an error when attempting to install Kafka integration.

```auto
Error installing kafka 1.7.0: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded

```

Evidently, the new limit (set to 21) doesn't seem to be recognized.

In an attempt to resolve this, I tried to apply the same for the @custom components specifically for the Kafka integration, but unfortunately, the error persists.

Could you provide guidance on how to globally update this setting for all new indices in the cluster? Your expertise and help would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![Julia\_Bardi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julia_bardi/32/79463_2.png) [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)\
**Post date:** [July 27, 2023, 7:47am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/5 "2023-07-27T07:47:18Z")

</div>

Could you share the request that you tried to update `.fleet_globals-1` and what is in the current settings if you query it?

I tried to set it and seems to work.

```auto
PUT _component_template/.fleet_globals-1
{
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "dimension_fields": {
            "limit": 21
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Moaath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moaath/32/121792_2.png) [@Moaath](https://discuss.elastic.co/u/Moaath)\
**Post date:** [July 27, 2023, 8:42am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/6 "2023-07-27T08:42:06Z")

</div>

I updated the settings using the 'index management' UI in kibana. Kindly find the current settings for .fleet\_globals-1:

```auto
GET _component_template/.fleet_globals-1
{
  "component_templates": [
    {
      "name": ".fleet_globals-1",
      "component_template": {
        "template": {
          "settings": {
            "index": {
              "mapping": {
                "dimension_fields": {
                  "limit": "21"
                }
              }
            }
          },
          "mappings": {
            "_meta": {
              "managed_by": "fleet",
              "managed": true
            },
            "dynamic_templates": [
              {
                "strings_as_keyword": {
                  "mapping": {
                    "ignore_above": 1024,
                    "type": "keyword"
                  },
                  "match_mapping_type": "string"
                }
              }
            ],
            "date_detection": false
          }
        },
        "_meta": {
          "managed_by": "fleet",
          "managed": true
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Julia\_Bardi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julia_bardi/32/79463_2.png) [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)\
**Post date:** [July 27, 2023, 9:40am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/7 "2023-07-27T09:40:41Z")

</div>

Okay, this looks good.  
Could you share the full error message from kibana logs when you are trying to install the Kafka package?

---

<div class="post-metadata">

**Author:** ![Moaath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moaath/32/121792_2.png) [@Moaath](https://discuss.elastic.co/u/Moaath)\
**Post date:** [July 27, 2023, 11:11am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/8 "2023-07-27T11:11:37Z")

</div>

kindly find the error logs below:

```auto
[2023-07-27T11:09:54.138+00:00][WARN][plugins.fleet] Failure to install package [kafka]: [ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded]
[2023-07-27T11:09:54.139+00:00][ERROR][plugins.fleet] uninstalling kafka-1.7.0 after error installing: [ResponseError: illegal_argument_exception: [illegal_argument_exception] Reason: Limit of total dimension fields [16] has been exceeded]
[2023-07-27T11:09:55.299+00:00][ERROR][plugins.fleet] Error: error deleting component template metrics-kafka.broker@package
    at deleteComponentTemplate (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.js:211:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at deleteAssets (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.js:177:5)
    at removeInstallation (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/remove.js:67:3)
    at handleInstallPackageFailure (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:203:7)
    at /usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:389:7
    at installPackageFromRegistry (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:358:12)
    at installPackage (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:577:22)
    at ensureInstalledPackage (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:138:25)
    at PackagePolicyClientWithAuthz.create (/usr/share/kibana/x-pack/plugins/fleet/server/services/package_policy.js:115:9)
    at createPackagePolicyHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/package_policy/handlers.js:241:27)
    at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:163:30)
    at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/target_node/src/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)
[2023-07-27T11:09:55.719+00:00][ERROR][plugins.fleet] failed to uninstall or rollback package after installation error Error: Saved object [epm-packages/kafka] not found

```

---

<div class="post-metadata">

**Author:** ![Julia\_Bardi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julia_bardi/32/79463_2.png) [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)\
**Post date:** [July 27, 2023, 12:41pm UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/9 "2023-07-27T12:41:46Z")

</div>

Unfortunately the logs don't have more information, this error happens when Fleet tries to rollback the installation of kafka by deleting the installed component templates.

From what I saw in my local testing, the stream `metrics-kafka.consumergroup` has more than 16 dimension fields.

One more thing to try is uninstall Kafka completely and try to install again, the `.fleet_globals-1` component template setting should be enough.  
Commands from console:

```auto
DELETE kbn:/api/fleet/epm/packages/kafka/1.7.0
{
  "force": true
}

POST kbn:/api/fleet/epm/packages/kafka/1.7.0
{
  "force": true
}

```

If this doesn't work, you could try to upgrade the stack to the latest version to see if it helps.

---

<div class="post-metadata">

**Author:** ![Moaath](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moaath/32/121792_2.png) [@Moaath](https://discuss.elastic.co/u/Moaath)\
**Post date:** [July 30, 2023, 6:46am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/10 "2023-07-30T06:46:29Z")

</div>

Thank you, Júlia. Unfortunately, the solution didn't work for me. For now, I will use the filebeat option. I will try integrating the Elastic Agent again after upgrading the stack.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2023, 6:46am UTC](https://discuss.elastic.co/t/kafka-integration-installation-error-limit-of-total-dimension-fields-16-has-been-exceeded/339052/11 "2023-08-27T06:46:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.