Es7.16, two indexes have same field cost conflicting type?

Elastic Stack Elasticsearch
1

I have two indexes.

PUT /test1
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "name": { "type": "text" }
    }
  }
}

PUT /test2
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "name": { "type": "integer" }
    }
  }
}

Both indexes have the field "name", but one is an integer and the other is a text.
When querying data from this two indexes, data went wrong.
Besides, kibana said "This field name has a type conflict across indices".
However, I have seen this description in reference "Indices are completely independent of each other and so there will be no conflict of field types between indices."[Removal of mapping types | Elasticsearch Guide [7.16] | Elastic]。
So in my opinion, my case should not cause conflict or query error.
Do I misunderstand or miss something???
And my current solution is just add some prefix before the same field, like making the "name" into "index1_name" and "index2_name". Is this a proper solution??

2

I suppose you created index pattern like "test*" in kibana and use it in Discover or somewhere. As the index pattern include both test1 and test2 index, it caused field type confliction for query.

The discription about confliction in Elasticsearch is just about index itself and maybe confliction will occur when query on such indices simultaneously using wildcard, alias or something.

2 Likes
3

Yes, that is correct @Tomo_M

1 Like
4

Thanks for your reply.
Yes, I created an index pattern like "test*" in kibana. And I do need to query on such indices simultaneously. In my case, I want to query among all fields (maybe 10k+ fields in 1-2k indices) using query string like:

GET /_search
{
  "query": {
    "query_string": {
      "query": an integer(1) or a text("helloworld")
    }
  }
}

And to avoid the field confliction, my current solution is just add some prefix before the same field, like making the "name" into "index1_name" and "index2_name". That's the reason I got 10k fields totally.

Am I using Elasticsearch in a proper way?

1 Like
5

Thanks for your reply.
Yes, I created an index pattern like "test*" in kibana. And I do need to query on such indices simultaneously. In my case, I want to query among all fields (maybe 10k+ fields in 1-2k indices) using query string like:

GET /_search
{
  "query": {
    "query_string": {
      "query": an integer(1) or a text("helloworld")
    }
  }
}

And to avoid the field confliction, my current solution is just add some prefix before the same field, like making the "name" into "index1_name" and "index2_name". That's the reason I got 10k fields totally.

Am I using Elasticsearch in a proper way?

6

This is open ended question. until you provide your proper use case no one will know what to do.
what your data looks like? what are you trying to achieve from it etc...

8

Thanks for your reply.
I'm trying to describe my situation in a short way.
My original data is made of hundreds of excel files. Each index in my Elasticsearch represents a sheet in excel. And fields in one index represent columns in one sheet.
Now in my Elasticsearch, I got 1500 indices (1500 sheets in my excel files), and each index has nearly 15 fields (nearly 15 columns in one sheet). So, therea are 20000+ fields totally. And different index may have 100~100000 documents.

And my query case is quite simple.

  1. Full-text searching. I hope I can simply input an integer or text (which means I don't know what field the integer/text is related to) and Elasticsearch returns to the related documents from different indices.
  2. Finding exact values. I also hope that I can explicitly query exact data from Elasticsearch. For example, I want to query the documents in index_1, which has the property index_1.id equals 1 (but the key "id" maybe a text in other index, that's why I add a prefix before "id").

And both queries should cost less than one second.

Currently, Elasticsearch can fullfill my case. But I am not sure whether my use case is good.
The reason I worried is that there are 1500 indices, 20000 fields and millions of documents in my Elasticsearch. I do not have the knowledge about whether Elasticsearch would have a good performance in my use case.

9

Why you are using integer field? Keyword family looks better for exact match even if it is constructed from digits.

11

I read the documents about keyword family, and find it may benefit my case. I will take your advice. Thank you! :+1:

But changing some "integer" fields into "keyword" fields can't decreate the number of fields in my es. Because I still need to do full-text query in text fields, do range query in integer fields(except IDs).

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.