Discover does not show any data for indices with _source disabled

Elastic Stack Kibana
1

Hello,

I disabled the _source field on a couple of indices yesterday and today I noticed that I can not see anything from those indices on Discover.

I can filter on values and fields, but everything is empty on Kibana Discover and I'm not sure why, should Kibana fetch the data using the fields API?

2

What version? and did you change the setting in Kibana Advanced settings? The setting directs to use source vs field for Discover?

3

I'm on 8.10.2

I found only 2 settings related to _source, one under General, the metaFields which is a text field with the following values: _source, _id, _index, _score

I removed _source from this list, but no change.

The other setting is under Discover, discover:searchFieldsFromSource, which was turned off, so I would expect Kibana to get the fields using the Fields API.

If there are any other setting, the description is not clear enough.

Since I'm using daily indices, I needed to rollback because we need to be able to see the data on Discover and we do not need the _source field for a couple of indices.

What else should I check to be able to remove the _source fields from some indices but still see them on Discover?

Everything else worked, I was able to filter using values on the fields, the alerts on the data worked and the dashboards also worked, but We couldn't see the data on Discover.

The following screenshots can give an example of what we had.

discover-01
discover-011774×676 86.7 KB

discover-02
discover-021184×576 122 KB

discover-03
discover-031720×662 108 KB

4

Hi @leandrojmp

Interesting, I just reproduced this ....

Let me poke internally; that is not the behavior I expected.

5

Oh, thanks!

At least I'm not crazy and something is indeed not right :smile:

6

Still checking may be a bit ... but it does work with _synthetic source

Screenshot 2023-10-25 at 8.55.34 AM
Screenshot 2023-10-25 at 8.55.34 AM2828×1616 355 KB

7

If I'm not wrong, synthetic source would reconstruct the source on the fly, right?

Also, I think this is still imited to TSDB indices, which is not the case, I'm also have some mappings that doesn't work with synthetic if I'm not wrong.

Should I open an issue anywhere or just wait to see if you can get more information about this?

8

Sort of...

Synthetic _source is Generally Available only for TSDB indices (indices that have index.mode set to time_series ). For other indices synthetic _source is in technical preview. Features in technical preview may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

I have a question in... on expected Discover behavior with _source disabled but feel free to open a ticket.

1 Like
9

Thanks!

I've opened an issue on github.

10

From Engineering:

The fields API in ES uses the _source field under the hood, so disabling _source will make it so the fields aren't returned (even when using the fields API)

From https://www.elastic.co/guide/en/elasticsearch/reference/8.10/search-fields.html:

The fields option returns values in the way that matches how Elasticsearch indexes them. For standard fields, this means that the fields option looks in _source to find the values, then parses and formats them using the mappings. Selected fields that can’t be found in _source are skipped.

Stephen:

Huh.... so that is totally not what I expected in Discover... When I see the word fields I always think about the actual fields, not the _source... I (and I suspect others) did not read the docs closely I always thought the values were actually pulled from the doc_values etc..So I guess I am hearing that Discover is working as designed, but it is pretty much not of much value if _source is disabled? I guess I was basically expecting Discover to work like it does with Synthetic _source

Engineering:

Right, I don't think Discover supports _source being disabled. We should definitely improve the experience to at least let you know something is wrong

11

This is a little confusing and not easy to find, or even present, on the documentation.

On Kibana you have this setting:

Screenshot from 2023-10-25 14-06-25
Screenshot from 2023-10-25 14-06-251375×190 20 KB

Read fields from _source
When enabled will load documents directly from _source. This is soon going to be deprecated. When disabled, will retrieve fields via the new Fields API in the high-level search service.

Reading this I assumed that the Fields API is unrelated to the presence of the _source field, which seems not to be the case from what you shared.

But the main issue I think is that the documentation about disabling the _source field does not mention Kibana.

I think that this can be improved adding an information that the _source field is required to show the data on Kibana Discover and the user will decide from this.

As a similar example, a couple of time ago I discovered that disabling expensive queries on Elasticsearch breaks Kibana Alerting system, and this was also not documented at the time.

So, I will need to have _source enabled and find other ways to save some space.

Thanks!

12

@leandrojmp I passed your feedback on directly.

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.