Mapping question

Elastic Stack Elasticsearch
1

I have old index hundreds of them which has following in pattern.

image
image1808×130 5.93 KB

I have another index created on different test cluster. which had this place dynamically.

image
image1675×132 5.98 KB

What I want is to add this host.name.keyword in previous index pattern.
I understand that I must have messed up something when I did create first set of index and patterns.

how do I add host.name.keyword now to previous pattern?

2

Actually your original / top example is more correct.

The second example is a default mapping meaning you did not define it ...

I would suggest to fix the test cluster.

You can't change the mapping on the 100s of existing indices

3

but it says it would be wise to use .keyword for aggregation as it is more optimized.

and I have only four of such field that needs to be changed.

4

Yes your top one IS a keyword

image
image1380×98 6.15 KB

Exactly

A keyword does not need to be named .keyword

It just needs to be a keyword type

Look at the mapping on the top index you will see here

5

ok. that make sense. but I don't know how it came up as host.name = keyword, this is using metricbeat template (7.12.0) I must have done something to be like that.

because second example is 7.17.1 metricbeat template.

6

one more question then
on second pic it shows
host.name = text,keyword. but I am not able to use that on viz it gives me this error

Reason
Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

but when I use host.name.keyword it works fine.

7

Most Likely you missed running setup... that is what happens when there is no mapping template

Whenever you see this

image
image1675×132 5.98 KB

That means the template is not applied... you did not run setup or you renamed the index to something the template does not match therefore the perdefined mapping is not applied and thus the default mapping IS applied (I show that at the bottom)

Yes this makes perfect sense...

host.name

Actually a multifield (2 fields in 1)

One is host.name is a text field which you can not aggregate on hence the error message above

the Second host.name.keyword is a keyword which you can use to aggregate on.

Do this... this is what is happening and you are seeing...

POST discuss-test/_doc
{
   "host.name" : "myhost"
}

GET discuss-test

{
  "discuss-test": {
    "aliases": {},
    "mappings": {
      "properties": {
        "host": {
          "properties": {
            "name": {
              "type": "text",  <!----- host.name of type text
              "fields": {
                "keyword": {
                  "type": "keyword", <!--- host.name.keyword of type keyword 
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }
...

What it should look like, and will look like with the proper templates / mapping applied

DELETE discuss-test

PUT discuss-test/
{
  "mappings": {
    "properties": {
      "host": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      }
    }
  }
}


POST discuss-test/_doc
{
   "host.name" : "myhost"
}

GET discuss-test

{
  "discuss-test": {
    "aliases": {},
    "mappings": {
      "properties": {
        "host": {
          "properties": {
            "name": {
              "type": "keyword"
            }
          }
        }
      }
    },
...
1 Like
8

This is great @stephenb I understand now what happened and how to fix/investigate further.

I will have to investigate and get this properly align.

basically my existing cluster have proper mapping. and I discover this while testing upgrade process on my test cluster.

I will have to find a way to duplicate that on test for proper testing.

9

Thanks again @stephenb
I was able to duplicate exactly what I had in production to test cluster.

what I had to do is host.name is not something that comes with metricbeat (fields.yml) hence once I put the default template which is composable index template. I added this

 "host": {
      "type": "object",
      "properties": {
        "name": {
          "type": "keyword"
        }
      }
    }

Actually I had define 10 such field for our use case. and they where all under object "host, system, processes"

I tested out with one object like host.name and it works now.

10

Hmmm... it should be

host.hostname

Hostname of the host. It normally contains what the hostname command returns on the host machine.

type: keyword

host.name

Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.

type: keyword

11

something wrong then because if I leave is as is, it creates exactly like my pic2.

but if I remove that multi level part from it. it does creates only one "host.name" as keyword

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.