## Summary
Fixes #101232
This PR adds support for the `ignored_field_value…s` API added in https://github.com/elastic/elasticsearch/issues/74121. This allows now seeing values which were ignored e.g. because they were too large or too long in Discover. We track this as a bug fix, since this is how the old _source implementation in Discover worked and was often perceived as a regression bug when switching to the new fields API implementation in Discover. Those values should now show back in Discover (and CSV reporting).
Ignored field values will show a warning now in Discover and filtering will be disabled on them:
**Limitations**:
* ignored values inside nested fields won't be shown by this PR. We decided not to support them by the API, since we don't have the knowledge anymore to know from which "object" inside the nested field the ignored value came, thus we can't reasonably merge them back together.
* We decided not to support showing those values if you have a "object column" in your discover table (e.g. `object` as a column with belows test data). You can't run with the fields API anymore in the situation where you can add object columns, thus we consider them to only be there for backwards compatibility with old saved searches, and won't add this functionality to them.
### Content of this PR
* Enables merging `ignored_field_values` in the `flattenHit` implementation in tabifyDocs (the one that should be used nowadays) behind a parameter. I think the default behavior should stay ignoring these values.
* Change the behavior we format whole documents in Discover to use `flattenField` instead of `formatHit` (which does not support that parameter yet).
* Change Discover logic to no longer use the _source field formatter at all but have all the logic inside it's own plugin. Since there are no further consumers for this "pretty rendered _source" outside Discover, we could simplify the _source field formatter to a simple `JSON.stringify` (to help get rid of `formatHit`).
* Remove `formatHit` and `formatField` completely from DataViews, since Discover and the old _source field formatter were the last consumers of this.
* Remove the `indexPattern` parameter from field formatters, since it was only used by the _source field formatter.
For more details please have a look at my inline comments on this PR and the comments I added to the code.
### How to test?
You will require some field values which are ignored either because they are too long or they are malformed. I've used the following commands to get different types of `ignore_above` ignored data:
<details>
<summary>Console commands for inserting data</summary>
```
PUT discover-test
{
"mappings": {
"properties": {
"number": {
"type": "long"
},
"text": {
"type": "keyword"
},
"short": {
"type": "keyword",
"ignore_above": 8
},
"nested": {
"type": "nested",
"properties": {
"short": {
"type": "keyword",
"ignore_above": 8
},
"more": {
"type": "keyword"
}
}
},
"object": {
"type": "object",
"properties": {
"short": {
"type": "keyword",
"ignore_above": 8
},
"more": {
"type": "keyword"
}
}
}
}
}
}
PUT discover-test/_doc/1?refresh=true
{
"number": 123,
"text": "test foo bar",
"short": ["only", "valid"],
"nested": [
{ "short": ["only", "valid"], "more": ["some", "more"] },
{ "short": ["too long value"], "more": ["some", "more"] }
],
"object": [
{ "short": ["only", "valid"], "more": ["some", "more"] },
{ "short": ["too long value"], "more": ["some", "more"] }
]
}
PUT discover-test/_doc/2?refresh=true
{
"number": [123],
"text": ["test foo bar"],
"short": "valid",
"nested": { "short": ["too long value"], "more": ["some", "more"] },
"object": { "short": ["too long value"], "more": ["some", "more"] }
}
PUT discover-test/_doc/3?refresh=true
{
"number": [],
"text": [null, null],
"short": "invalid one",
"nested": [
{ "short": "this one too long too" },
{ "short": "too long value" }
],
"object": [
{ "short": "this one too long too" },
{ "short": "too long value" }
]
}
PUT discover-test/_doc/4?refresh=true
{
"number": null,
"text": [""],
"short": ["mixed", "valid and invalid"],
"nested": [
{ "short": ["mixed", "with some valid some invalid"], "more": ["some", "more"] },
{ "short": ["mixed", "with some valid some invalid"], "more": ["some", "more"] }
],
"object": [
{ "short": ["mixed", "with some valid some invalid"], "more": ["some", "more"] },
{ "short": ["mixed", "with some valid some invalid"], "more": ["some", "more"] }
]
}
PUT discover-test/_doc/5?refresh=true
{
"short": """[2021-10-14T18:45:33.441+02:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 132a44cb-715c-4b78-9e9a-3a8b56411d4b
[2021-10-14T18:45:33.512+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2021-10-14T18:45:33.524+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2021-10-14T18:45:33.567+02:00][INFO ][plugins.ruleRegistry] Write is disabled; not installing common resources shared between all indices
[2021-10-14T18:45:33.835+02:00][INFO ][plugins.reporting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Arch Linux Rolling OS. Automatically enabling Chromium sandbox.
[2021-10-14T18:45:33.848+02:00][INFO ][plugins.ruleRegistry] Write is disabled; not installing resources for index .alerts-observability.uptime.alerts
[2021-10-14T18:45:33.848+02:00][INFO ][plugins.ruleRegistry] Write is disabled; not installing resources for index .alerts-observability.logs.alerts
[2021-10-14T18:45:33.848+02:00][INFO ][plugins.ruleRegistry] Write is disabled; not installing resources for index .alerts-observability.metrics.alerts
[2021-10-14T18:45:33.849+02:00][INFO ][plugins.ruleRegistry] Write is disabled; not installing resources for index .alerts-observability.apm.alerts
[2021-10-14T18:45:33.943+02:00][INFO ][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
[2021-10-14T18:45:33.943+02:00][INFO ][savedobjects-service] Starting saved objects migrations
[2021-10-14T18:45:34.028+02:00][INFO ][savedobjects-service] [.kibana] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 77ms.
[2021-10-14T18:45:34.029+02:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 77ms.
[2021-10-14T18:45:34.095+02:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 67ms.
[2021-10-14T18:45:34.096+02:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 67ms.
[2021-10-14T18:45:34.164+02:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 69ms.
[2021-10-14T18:45:34.169+02:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 73ms.
[2021-10-14T18:45:34.253+02:00][INFO ][savedobjects-service] [.kibana] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 89ms.
[2021-10-14T18:45:34.255+02:00][INFO ][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 86ms.
[2021-10-14T18:45:34.422+02:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 169ms.
[2021-10-14T18:45:34.423+02:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 168ms.
[2021-10-14T18:45:34.480+02:00][INFO ][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 57ms.
[2021-10-14T18:45:34.480+02:00][INFO ][savedobjects-service] [.kibana_task_manager] Migration completed after 528ms
[2021-10-14T18:45:34.532+02:00][INFO ][savedobjects-service] [.kibana] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 110ms.
[2021-10-14T18:45:34.533+02:00][INFO ][savedobjects-service] [.kibana] Migration completed after 582ms"""
}
```
</details>
### Out of scope
Out of scope of this PR (issues to be created):
- Better and longer texts for ignored values (added to https://github.com/elastic/kibana/issues/114271)
- Improve disabled filter for ignored values (better help tooltip (currently buttons are just gone) & make it work in data grid)
- Improve filter in sidebar for values that are ignored (somehow show a hint beside the filter)
- Add better warning icon (via EUI) (https://github.com/elastic/eui/issues/5274)
### Checklist
Delete any items that are not applicable to this PR.
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- ~[ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- ~[ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)~
- [x] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
