Index pattern not returning results

aaronloesattv · March 12, 2018, 8:33pm

We have a couple time series indexes that have some overlap in name that require a more-complex-than-standard index pattern in order to target a specific index. For example we have an index called this-is-my-index-2018.01 and one called this-is-my-index-also-2018.01. Creating an index pattern for the second index is easy, its just this-is-my-index-also-* and this works just fine, we only see data stored in the second index. However, when creating the index pattern for the first index, we used the pattern this-is-my-index-*,-*also* which should exclude data stored in the second index. When creating this pattern, Kibana lists the indexes that would match and i can see the second index is not included in this match. With the index pattern created, Kibana also displays the mapping informtaion. I can also use this index pattern in raw queries and get only the data stored in the first index. All this is what i expect. However, when i go to the discover page of Kibana and select this index pattern, no data is ever returned.

I know there is data in the index. Kibana was able to show me which indexes match my pattern. It displays the mapping information for the index pattern. However, i cannot seem to get any data to display in the dashboard view.

Is there a limitation with index patterns that i'm not aware of in Kibana or is this a bug?

Thank you.

Brandon_Kobel · March 12, 2018, 9:23pm

@aaronloesattv are you using X-Pack Security in Elasticsearch? Do you have the required permissions on the data indices to be able to search the data? Are you seeing any data? If you change your index-pattern to something like this-is-my-index-* do you see any data?

aaronloesattv · March 12, 2018, 9:56pm

@Brandon_Kobel , we do have xpack security and i am the superuser. I am able to get to the data if i create an index pattern like this-is-my-* and manually filter out the records that are in the un-desired index.

Brandon_Kobel · March 13, 2018, 11:48am

@aaronloesattv are you not seeing any results when searching using your ideal index pattern? If you open up your browser's dev tools, you should see a network request for the _msearch that we execute against Elasticsearch, if you execute this _msearch yourself do you see any results?

aaronloesattv · March 13, 2018, 2:43pm

@Brandon_Kobel Hope this is not too verbose.

Here is a compacted console output showing the two interactions. The first is the _msearch request and response for the index pattern this-is-my-index-*,-*also* showing no results. The second is the _msearch request and response for the index pattern this-is-my-index-* with the added query NOT _index: *also* added to remove records in the undesired index. You'll see that there are results in the second response that are only in the desired index this-is-my-index-*.

Request

POST https://example.com:5601/elasticsearch/_msearch
{
  "index": [
    "this-is-my-index-*,-*also*"
  ],
  "ignore_unavailable": true,
  "preference": 1520949814359
}
{
  "version": true,
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30m",
        "time_zone": "America/Chicago",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    "@received_at",
    "@timestamp",
    "journal.source_realtime_timestamp",
    "log.@timestamp",
    "received_at"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1520864079230,
              "lte": 1520950479230,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

Response

{
  "responses": [
    {
      "took": 0,
      "timed_out": false,
      "_shards": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": 0,
        "max_score": 0.0,
        "hits": []
      },
      "status": 200
    }
  ]
}

and here is the request using the simplified index pattern this-is-my-index-* that matches data in both indexes but manually filtering out data from the undesired index:

Request

https://example.com:5601/elasticsearch/_msearch
{
  "index": [
    "this-is-my-index-*"
  ],
  "ignore_unavailable": true,
  "preference": 1520949814359
}
{
  "version": true,
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30m",
        "time_zone": "America/Chicago",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    "@received_at",
    "@timestamp",
    "journal.source_realtime_timestamp",
    "log.@timestamp"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "NOT _index: *also*",
            "analyze_wildcard": true,
            "default_field": "*"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1520864269438,
              "lte": 1520950669438,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

Response

{
  "responses": [
    {
      "took": 2593,
      "timed_out": false,
      "_shards": {
        "total": 12,
        "successful": 12,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": 14888525,
        "max_score": null,
        "hits": [
          ... snip ...
          {
            "_index": "this-is-my-index-2018.03",
            "_type": "doc",
            "_id": "45G5H2IBQ-c6TSCvU0qq",
            "_version": 1,
            "_score": null,
            "_source": {
              ... snip ...
            },
            "fields": {
              "@received_at": [
                "2018-03-13T14:17:42.011Z"
              ],
              "@timestamp": [
                "2018-03-13T14:17:39.257Z"
              ],
              "log.@timestamp": [
                "2018-03-13T14:17:39.257Z"
              ]
            },
            "sort": [
              1520950659257
            ]
          },
          ... snip ...
        ]
      },
      "aggregations": {
        "2": {
          "buckets": [
            {
              "key_as_string": "2018-03-12T09:00:00.000-05:00",
              "key": 1520863200000,
              "doc_count": 125021
            },
            ... snip ...
          ]
        }
      },
      "status": 200
    }
  ]
}

aaronloesattv · March 13, 2018, 3:08pm

@Brandon_Kobel i dont know if its more helpful but i've clearly been dancing around what the actual index names are as to remove any confusing and unhelpful information but the index names are actually more like the following:

log-our-application-name-2018.01
log-our-application-name-2018.02
log-our-application-name-2018.03
log-our-application-name-docker-swarm-2018.01
log-our-application-name-docker-swarm-2018.02
log-our-application-name-docker-swarm-2018.03

and the index pattern we use to get the docker-swarm variant of our logs is log-our-application-name-docker-swarm-* and the index pattern in question is actually log-our-application-name-*,-*docker-swarm* to get the non docker-swarm variant of our logs

Brandon_Kobel · March 14, 2018, 12:17pm

Hey @aaronloesattv, you've uncovered some interesting behavior. I created https://github.com/elastic/kibana/issues/17147 to track what you're seeing, and I've pinged the internal teams with regard to the issue to see what's happening here.

aaronloesattv · March 14, 2018, 2:42pm

thank you for letting me know.

Brandon_Kobel · March 14, 2018, 8:26pm

@aaronloesattv if you haven't been following along in that GitHub ticket, the ability to create index patterns with a "," in them isn't something that Kibana used to support. It became possible to specify an index pattern with a "," via the Management page in 6.1 (which was removed in 6.2) but they didn't work throughout Kibana. We're looking into what should be done to make the "," index patterns work fully. I'd recommend subscribing to the aforementioned GitHub issue to track the progress.

system · April 11, 2018, 8:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.