Elasticsearch cannot access keys although they are available

Hello,

recently I have been running into an issue with elastic.

I have a single node Elasticsearch-Container running in Docker.

Sometimes it seems that Elasticsearch is not running as it should so I have tried to debug it using kibana.

In case (case 1) it runs properly, the following queue

GET /trend_meta/_search

{
    "query": {
        "constant_score": {
            "filter": {
                "term": {
                    "id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f"
                }
            }
        }
    }
}

results in this response:

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
        },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
            },
        "max_score": 1,
        "hits": [
            {
                "_index": "trend_meta",
                "_id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f",
                "_score": 1,
                "_source": {
                    "id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f",
                    "liveTime": 60,
                    "legend": {
                        "legendType": "Top",
                        "column": 3
                        },
                    "labelIt": [
                        {
                        "languageTag": "en",
                        "text": "Trend 01"
                        }
                    ]
                }
            }
        ]
    }
}

if it is not working properly (case 2), the response is:

{
"took": 0,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
    },
"hits": {
    "total": {
        "value": 0,
        "relation": "eq"
        },
    "max_score": null,
    "hits": []
    }
}

If I would use "_id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f" instead of "id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f" in the search query, the response would be as expected (like the response of case 1), no matter if the Elasticsearch is running well or with issues.

The key _id is the id given from Elasticsearch. The key id is a key I defined.

GET /_template/trend_meta

response:

{
"trend_meta": {
    "order": 0,
    "index_patterns": [
        "trend_meta"
        ],
    "settings": {},
    "mappings": {
        "properties": {
            "id": {
                "type": "keyword"
                }
            }
        }, "aliases": {}
    }
}

My problem is that my code does not allow me to use the key _id instead of id. Somehow there seem to be cases when the key id is available but not accessible.

Have you run into the same issue before and have you been able solve it?

Welcome!

May be the document is not searchable yet because the index has not been refreshed?

You can call:

POST trend_meta/_refresh

Before searching and check if this works.

If not, can you run:

GET trend_meta/_doc/trend9d27baa7-7dbe-4448-891a-814cab9e805f

Where trend9d27baa7-7dbe-4448-891a-814cab9e805f is the id you are searching for.

1 Like

Hello,
thank you for your reply and the help you have offered.

I did run

POST trend_meta/_refresh

and received

{
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  }
}

as an answer. The query

GET /trend_meta/_search
{
  "query": {
    "constant_score": {
        "filter": {
                    "term": {
                        "id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f"
                }
              
        }
  }
}
}

is still not working if I use “id” instead of “_id”.

Running

GET trend_meta/_doc/trend9d27baa7-7dbe-4448-891a-814cab9e805f

results in this answer:

{
  "_index": "trend_meta",
  "_id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f",
  "_version": 2,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f",
    "liveTime": 60,
  }
}

I still have no idea why the key “id” is available but not accessible.

If I use the key "liveTime":

GET /trend_meta/_search
{
  "query": {
    "constant_score": {
        "filter": {
                    "term": {
                        "liveTime": 60
                }
              
        }
  }
}
}

for my search-query it works just as fine as using the underline id: "_id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f".

Can you please use the get mapping API and post the result here so we can ensure the index template you listed has been correctly applied?

If the template has not been successfully applied I suspect you may have default mappings, which means using the keyword subfield might work:

GET /trend_meta/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "term": {
          "id.keyword": "trend9d27baa7-7dbe-4448-891a-814cab9e805f"}
        }
      }
    }
  }
}

Thank you for replying.
Using the get mapping API:

GET /trend_meta/_mapping

I get the following (full) result:

{
  "trend_meta": {
    "mappings": {
      "properties": {
        "id": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "labelIt": {
          "properties": {
            "languageTag": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "text": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "legend": {
          "properties": {
            "column": {
              "type": "long"
            },
            "legendType": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "liveTime": {
          "type": "long"
        },
        "scales": {
          "properties": {
            "decimalPlaces": {
              "type": "long"
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "max": {
              "type": "long"
            },
            "maxAuto": {
              "type": "boolean"
            },
            "min": {
              "type": "long"
            },
            "minAuto": {
              "type": "boolean"
            }
          }
        },
        "trendEntryMetas": {
          "properties": {
            "attributeAsLabel": {
              "type": "boolean"
            },
            "attributeInfoTexts": {
              "properties": {
                "languageTag": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "text": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "domainName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "entryId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "entryType": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "labelIt": {
              "properties": {
                "languageTag": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "text": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "legend": {
              "type": "boolean"
            },
            "liveValue": {
              "type": "boolean"
            },
            "order": {
              "type": "long"
            },
            "scaleId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "serviceName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "style": {
              "properties": {
                "color": {
                  "type": "long"
                },
                "dasharray": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "fill": {
                  "type": "long"
                },
                "fontColor": {
                  "type": "long"
                },
                "width": {
                  "type": "long"
                }
              }
            },
            "trendEntryCalculate": {
              "properties": {
                "offset": {
                  "type": "long"
                },
                "scaleFactor": {
                  "type": "long"
                }
              }
            }
          }
        }
      }
    }
  }
}

using "id.keyword": "trend9d27baa7-7dbe-4448-891a-814cab9e805f" in the search query results in:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "trend_meta",
        "_id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f",
        "_score": 1,
        "_source": {
          "id": "trend9d27baa7-7dbe-4448-891a-814cab9e805f",
          "liveTime": 60,
          "legend": {
            "legendType": "Top",
            "column": 3
          },
          "labelIt": [
            {
              "languageTag": "en",
              "text": "Trend 01"
            }
          ],
          "scales": [
            {
              "id": "default",
              "labelIt": [],
              "decimalPlaces": 1,
              "minAuto": false,
              "min": 0,
              "maxAuto": false,
              "max": 160
            }
          ],
          "trendEntryMetas": [
            {
              "liveValue": false,
              "trendEntryCalculate": {
                "offset": 0,
                "scaleFactor": 1
              },
              "entryType": "META_VALUE",
              "entryId": "value-dp-1311655834a-1558567263trend9d27baa7-7dbe-4448-891a-814cab9e805f",
              "domainName": "ServerB",
              "serviceName": "AA",
              "style": {
                "color": 16711680,
                "width": 2,
                "dasharray": "",
                "fill": 100,
                "fontColor": 0
              },
              "scaleId": "default",
              "legend": true,
              "attributeAsLabel": true,
              "order": 1,
              "labelIt": [
                {
                  "languageTag": "en",
                  "text": "A_2"
                }
              ],
              "attributeInfoTexts": [
                {
                  "languageTag": "en",
                  "text": "A-Val 2"
                }
              ]
            },
            {
              "liveValue": false,
              "trendEntryCalculate": {
                "offset": 0,
                "scaleFactor": 1
              },
              "entryType": "META_VALUE",
              "entryId": "value-dp-1311655835a-1558567264trend9d27baa7-7dbe-4448-891a-814cab9e805f",
              "domainName": "ServerB",
              "serviceName": "AA",
              "style": {
                "color": 16753920,
                "width": 2,
                "dasharray": "",
                "fill": 100,
                "fontColor": 0
              },
              "scaleId": "default",
              "legend": true,
              "attributeAsLabel": true,
              "order": 0,
              "labelIt": [
                {
                  "languageTag": "en",
                  "text": "A_1"
                }
              ],
              "attributeInfoTexts": [
                {
                  "languageTag": "en",
                  "text": "A-Val 1"
                }
              ]
            },
            {
              "entryType": "META_LEVEL",
              "entryId": "level-dp-1311655832a-1558567263trend9d27baa7-7dbe-4448-891a-814cab9e805f",
              "domainName": "ServerB",
              "serviceName": "AA",
              "style": {
                "color": 9127187,
                "width": 2,
                "dasharray": "",
                "fill": 100,
                "fontColor": 0
              },
              "scaleId": "default",
              "legend": true,
              "attributeAsLabel": true,
              "order": 1,
              "labelIt": [
                {
                  "languageTag": "en",
                  "text": "A_4"
                }
              ],
              "attributeInfoTexts": [
                {
                  "languageTag": "en",
                  "text": "A-Val 2"
                }
              ]
            },
            {
              "entryType": "META_LEVEL",
              "entryId": "level-dp-1311655833a-1558567264trend9d27baa7-7dbe-4448-891a-814cab9e805f",
              "domainName": "ServerB",
              "serviceName": "AA",
              "style": {
                "color": 32768,
                "width": 2,
                "dasharray": "",
                "fill": 100,
                "fontColor": 0
              },
              "scaleId": "default",
              "legend": true,
              "attributeAsLabel": true,
              "order": 0,
              "labelIt": [
                {
                  "languageTag": "en",
                  "text": "A_3"
                }
              ],
              "attributeInfoTexts": [
                {
                  "languageTag": "en",
                  "text": "A-Val 1"
                }
              ]
            }
          ]
        }
      }
    ]
  }
}

It looks like the index template you listed was not applied when the index was created, so it seems you indeed have default mappings applied. Note that index templates are only applied when the index is first created.

1 Like

Thank you again for your answer.

May I ask how you can tell if the template is not applied?

In the project that is working it looks like this:
GET /trend_meta/_mapping :

{
  "trend_meta": {
    "mappings": {
      "properties": {
        "id": {
          "type": "keyword"
        },
        "labelIt": {
          "properties": {
            "languageTag": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "text": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "legend": {
          "properties": {
            "column": {
              "type": "long"
            },
            "legendType": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "liveTime": {
          "type": "long"
        },
        "scales": {
          "properties": {
            "decimalPlaces": {
              "type": "long"
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "max": {
              "type": "long"
            },
            "maxAuto": {
              "type": "boolean"
            },
            "min": {
              "type": "long"
            },
            "minAuto": {
              "type": "boolean"
            }
          }
        },
        "trendEntryMetas": {
          "properties": {
            "attributeAsLabel": {
              "type": "boolean"
            },
            "attributeInfoTexts": {
              "properties": {
                "languageTag": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "text": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "domainName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "entryId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "entryType": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "labelIt": {
              "properties": {
                "languageTag": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "text": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "legend": {
              "type": "boolean"
            },
            "liveValue": {
              "type": "boolean"
            },
            "order": {
              "type": "long"
            },
            "scaleId": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "serviceName": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "style": {
              "properties": {
                "color": {
                  "type": "long"
                },
                "dasharray": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "fill": {
                  "type": "long"
                },
                "fontColor": {
                  "type": "long"
                },
                "width": {
                  "type": "long"
                }
              }
            },
            "trendEntryCalculate": {
              "properties": {
                "offset": {
                  "type": "long"
                },
                "scaleFactor": {
                  "type": "long"
                }
              }
            }
          }
        }
      }
    }
  }
}

and the request

GET /trend_meta/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "term": {
          "id.keyword": "trend9d27baa7-7dbe-4448-891a-814cab9e805f"}
        }
      }
    }
  }
}

receives:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

I use two containers. One with elastic and one with the project. It seems to me that the order in which they start determins if the project will work or not.

It seems like the mapping changed? The second example is what should be there if the index template was applied. Did you delete and recreate the index?

I have two projects. One that is not working and one that works.
The first example is the one that is not woirking and the second example is the one that works.

The difference between the project that is not working and the one that is working are the following lines:

"type": "text",
"fields": {
  "keyword": {
  "type": "keyword",
  "ignore_above": 256

Are these lines part of default mappings? And is it how you can detect if the template was applied correctly or not?

Yes, that is the default mapping for a text field.

If the template was applied the mappings of the index would be aligned with the index template, which they in your first example are not.

Thank you so much for taking the time to help me understand that issue. I did learn alot from your answers :slightly_smiling_face: