Can't view fields In Discovery or via query even though they're available in Kibana visualizations

Elastic Stack Elasticsearch
1

I have a few indexes, defined by a template, and each index is distinguished by year. When querying the current years index (previous years are fine), I get no fields to return any values. When looking at visualizations in Kibana, the fields are there and can be calculated.

I've also refreshed my index in Kibana and there are no mapping conflicts. Any thoughts on something I may be missing. I feel like it should be obvious, but I'm getting hung up on it.

Here is the template:

{
      "cr-ops-wrike*" : {
    "order" : 0,
    "index_patterns" : [
      "cr-ops-wrike-*",
      "cr-ops-test-wrike-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1"
      }
    },
    "mappings" : {
      "_source" : {
        "enabled" : false
      },
      "properties" : {
        "timelog_hours_logged" : {
          "type" : "float"
        }
      }
    },
    "aliases" : { }
      }
}
2

Hello @schoffelman

Can you share the output of GET cr-ops-*wrike-* and an example of the query which doesn't return any value vs one which returns the results?

3

You bet. Here's the query:

GET /cr-ops-wrike-time-2020/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2020-03-01"
      }
    }
  }
}

And here is a sample result:

{
  "took" : 10,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "cr-ops-wrike-time-2020",
        "_type" : "_doc",
        "_id" : "3f6BwHABnH_-7kJihrsw",
        "_score" : 1.0
      },
4

The previous index looks like this:

Query:

GET /cr-ops-wrike-time-2019/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2020-02-01"
      }
    }
  }
}

Sample Result:

"hits" : [
      {
        "_index" : "reindexed-v7-cr-ops-wrike-time-2019",
        "_type" : "_doc",
        "_id" : "2heWDHABj8nezjVaIsOX",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-02-01T12:00:00Z",
          "task_id" : "IEABLR4CKQEXUMNV",
          "task_title" : "General Office",
          "task_created_date" : "2017-06-15T20:05:15Z",
          "task_updated_date" : "2019-09-03T13:57:36Z",
          "task_permalink" : "https://www.wrike.com/open.htm?id=**********",
          "task_status" : "Active",
          "client_folder_id" : "IEABLR4CI4EXULV7",
          "client_folder_name" : "[CR] General Ops",
          "timelog_id" : "IEABLR4CJQAE57VZ",
          "timelog_hours_logged" : 1,
          "timelog_comment" : "*******************",
          "timelog_created_date" : "2020-02-01T20:04:07Z",
          "timelog_updated_date" : "2020-02-01T20:04:07Z",
          "timelog_tracked_date" : "2020-02-01",
          "timelog_api_count" : 316,
          "meta_Client" : "**********",
          "user_id" : "KUADHE7B",
          "user_name" : "*****",
          "user_email" : "****@*******.com",
          "user_active" : 1,
          "user_production" : 1
        }
      },
5

I can't remember, but I may have created the template after the previous indexes were created, but before the 2020 one was populated. In that case, would the _source field be the culprit?

6

Thanks, can you run exactly the query above (so I can check the mappings)?

On this query:

  • GET /cr-ops-wrike-time-2020/_search you have an hit, but as you've disabled the _source thanks to the index template, the fields are not shown. You might be able to see the "docvalue_fields" for the fields where you have docvalues
  • GET /cr-ops-wrike-time-2019/_search you have few hits and you're able to see the _source.

This happens probably because you updated/added the index template after the creation of the 2019 index, but before the creation of the index 2020.

7

Yes, it can be the reason.
The problem now is _source is disabled so you cannot reindex.
Do you have the original data?

8

Yep. It's not too hard to reindex. I just wanted to know the problem before redoing some of it so I didn't run into that situation again.

Thanks for your help!

1 Like
9

Update the index template to remove _source (so it's enabled by default), remove the 2020 index and re-ingest and it should be fine :wink:

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.