Mapping conflict

[mb19 (maria)]

Hello everyone,

I am new to using the Elasticsearch and Kiabana stack, I do not currently have Logstash installed and would prefer not to install it.

I recently had this error:

I don't know how to fix it, I think I have some conceptual problems, I have tried creating a second index called "prueba" using the reindex API:

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "metrics-*"
  },
  "dest": {
    "index": "prueba"
  }
}

But I don't know how to proceed now.

• Would it be possible to simply fix the data type of that field?

• Would it be possible to delete the field or delete the metrics and start again from scratch? (since I am in a testing environment and the information is not relevant)

Although I would like to know how to solve it properly for when the environment is production

Thank you very much to all!

[dadoonet (David Pilato)]

Welcome!

The error message indicates that you have one of the index starting with metrics- which has a wrong mapping.
So you need to find first what is this index.

You can do that by comparing all the mappings (which might be a very manual task though). You can run the following command in the Dev Console:

GET /metrics-*/_mapping/field/host.ip

Once you know, you can:

• Remove the wrong index (but this will remove its data)
• Reindex the wrong index with the right mapping to a new index named like metrics-xxx-fixed and then remove the wrong index

But you also need to understand who or which tool created this wrong index. Was that a manual test you did? A bug?...

@mb19

Would it be possible to simply fix the data type of that field?

No. "Simply" would mean to reindex. Which is what I suggested and what you did somehow.

@mb19

Would it be possible to delete the field or delete the metrics and start again from scratch? (since I am in a testing environment and the information is not relevant)

Yes.

Hope this helps

[mb19 (maria)]

Hello! Thank you very much for the quick response,

About finding the conflicting index, sorry, I knew which but I forgot to mention it. Although I didn't really know how to use this information either, Kibana's own interface shows it here:

About the tool that created the wrong index, I don't know. From what I see in the image, I guess that it was the mysql integration.

I have a policy in one agent with these two integrations ( System Integration and MySQL Integration), and it was yesterday when I added the mysql one and when I saw the error.

But about reindexing, what I did was reindex all those indexes that started with "metrics-", right? So what should I do in this specific case is this?

POST _reindex?wait_for_completion=false
{
   "source": {
     "index": ".ds-metrics-mysql.performance-default-2024.01.11-000001"
   },
   "dest": {
     "index": ".ds-metrics-mysql.performance-default-2024.01.11-000001-FIXED"
   }
}

​

[dadoonet (David Pilato)]

Yes. Something like:

POST _reindex?wait_for_completion=false
{
   "source": {
     "index": ".ds-metrics-mysql.performance-default-2024.01.11-000001"
   },
   "dest": {
     "index": ".ds-metrics-mysql.performance-default-2024.01.11-000001-FIXED"
   }
}
DELETE .ds-metrics-mysql.performance-default-2024.01.11-000001

[mb19 (maria)]

Hello,

Unfortunately it hasn't worked, the console does not warn if the task has finished or not, I cannot find the new index either in "Index Management" or in "dataView/metrics-*"

Finally if I try to delete it by executing:

DELETE .ds-metrics-mysql.performance-default-2024.01.11-000001

I get:

index [.ds-metrics-mysql.performance-default-2024.01.11-000001] is the write index for data stream [metrics-mysql.performance-default] and cannot be deleted"

Then I have tried this;

DELETE /_data_stream/.ds-metrics-mysql.performance-default-2024.01.11-000001

But then I get:

data streams [.ds-metrics-mysql.performance-default-2024.01.11-000001] not found"

Although the initial error about the mapping conflict persists and so does the field with this entry (I have refreshed the page)

[dadoonet (David Pilato)]

wait_for_completion=false does not wait for reindexing to be finished.
So you need to either set it to true or use the tasks API.

As you said you don't need the old data, so remove everything and start again might be the easiest way.

GET /_data_stream/

will give you the exact name to use.

[mb19 (maria)]

Hello,

On the one hand, if I put "wait_for_completion=false", I get the following output:

On the other hand, maybe I'm not understanding something, the exact name is told to me by the conflicting field, anyway I used the command you suggested:

GET /_data_stream/

And I have obtained this (which matches the name seen in the first screenshot of the post)

{
      "name": "metrics-mysql.performance-default",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-metrics-mysql.performance-default-2024.01.11-000001",
          "index_uuid": "2XQCG80ySGWbW1UU51h2Bg",
          "prefer_ilm": true,
          "ilm_policy": "metrics",
          "managed_by": "Index Lifecycle Management"
        }
    ......

Therefore I still have the same problem if I run

DELETE /_data_stream/.ds-metrics-mysql.performance-default-2024.01.11-000001

This is the output:

{
  "error": {
    "root_cause": [
      {
        "type": "resource_not_found_exception",
        "reason": "data streams [.ds-metrics-mysql.performance-default-2024.01.11-000001] not found"
      }
    ],
    "type": "resource_not_found_exception",
    "reason": "data streams [.ds-metrics-mysql.performance-default-2024.01.11-000001] not found"
  },
  "status": 404
}

[mb19 (maria)]

Little update, I ran this and the error is gone although I assume the data too:

DELETE /_data_stream/metrics-mysql.performance-default

For future occasions I would like to know what I could do since now the information was not important but later it will be

Thank you so much!