tl:dr; Add a dedicated http REST CRUD API to manage Dashboards to support infra…-as-code use-cases.
-----
**Current approach: using the saved object API**
Dashboards can be programmatically managed with the [saved object API](https://www.elastic.co/guide/en/kibana/current/saved-objects-api.html). This is a general purpose API for all resource types in Kibana (e.g. Dashboards, individual visualizations, saved searches, etc...) supporting basic CRUD operations.
A typical dashboard would consist of some dashboard metadata (title, description, layout, ...), and a collection of panels, each with panel specific contents (either by value or reference).
For example:
- `optionsJson`: options specific to a dashboard (e.g. border-width, ...)
- `panelsJson`: each panel has a position and a content. The panel-content (`embeddableConfig`) is specific to the type of panel (e.g. a Lens visualization, a map, ...)
```
{
"id": "730ea5e4-dc12-4b1c-aee4-a6af849be9be",
"type": "dashboard",
"namespaces": [
"default"
],
"updated_at": "2024-01-08T22:30:30.879Z",
"created_at": "2024-01-08T22:30:30.879Z",
"version": "Wzg1LDdd",
"attributes": {
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"
},
"description": "",
"timeRestore": false,
"optionsJSON": "{\"useMargins\":true,\"syncColors\":false,\"syncCursor\":true,\"syncTooltips\":false,\"hidePanelTitles\":false}",
"panelsJSON": "[{\"type\":\"lens\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"47acda3e-9f92-4220-b314-8c36f31a1551\"},\"panelIndex\":\"47acda3e-9f92-4220-b314-8c36f31a1551\",\"embeddableConfig\":{\"attributes\":{\"title\":\"\",\"visualizationType\":\"lnsXY\",\"type\":\"lens\",\"references\":[{\"type\":\"index-pattern\",\"id\":\"01d64a72-a702-4a41-8ba3-b87d45c40814\",\"name\":\"indexpattern-datasource-layer-d92f3431-88f6-455c-aecb-bb7c3ff82692\"}],\"state\":{\"visualization\":{\"legend\":{\"isVisible\":true,\"position\":\"right\"},\"valueLabels\":\"hide\",\"fittingFunction\":\"None\",\"axisTitlesVisibilitySettings\":{\"x\":true,\"yLeft\":true,\"yRight\":true},\"tickLabelsVisibilitySettings\":{\"x\":true,\"yLeft\":true,\"yRight\":true},\"labelsOrientation\":{\"x\":0,\"yLeft\":0,\"yRight\":0},\"gridlinesVisibilitySettings\":{\"x\":true,\"yLeft\":true,\"yRight\":true},\"preferredSeriesType\":\"bar_stacked\",\"layers\":[{\"layerId\":\"d92f3431-88f6-455c-aecb-bb7c3ff82692\",\"accessors\":[\"393ddff7-a95a-4572-90a8-1b5c73da8111\"],\"position\":\"top\",\"seriesType\":\"bar_stacked\",\"showGridlines\":false,\"layerType\":\"data\",\"xAccessor\":\"82884702-45e2-4657-86e3-7710517d038f\"}]},\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filters\":[],\"datasourceStates\":{\"formBased\":{\"layers\":{\"d92f3431-88f6-455c-aecb-bb7c3ff82692\":{\"columns\":{\"82884702-45e2-4657-86e3-7710517d038f\":{\"label\":\"@timestamp\",\"dataType\":\"date\",\"operationType\":\"date_histogram\",\"sourceField\":\"@timestamp\",\"isBucketed\":true,\"scale\":\"interval\",\"params\":{\"interval\":\"auto\",\"includeEmptyRows\":true,\"dropPartials\":false}},\"393ddff7-a95a-4572-90a8-1b5c73da8111\":{\"label\":\"Count of records\",\"dataType\":\"number\",\"operationType\":\"count\",\"isBucketed\":false,\"scale\":\"ratio\",\"sourceField\":\"___records___\",\"params\":{\"emptyAsNull\":true}}},\"columnOrder\":[\"82884702-45e2-4657-86e3-7710517d038f\",\"393ddff7-a95a-4572-90a8-1b5c73da8111\"],\"incompleteColumns\":{},\"sampling\":1}}},\"indexpattern\":{\"layers\":{}},\"textBased\":{\"layers\":{}}},\"internalReferences\":[],\"adHocDataViews\":{}}},\"enhancements\":{}}}]",
"title": "mydashboard"
},
"references": [
{
"type": "index-pattern",
"id": "01d64a72-a702-4a41-8ba3-b87d45c40814",
"name": "47acda3e-9f92-4220-b314-8c36f31a1551:indexpattern-datasource-layer-d92f3431-88f6-455c-aecb-bb7c3ff82692"
}
],
"managed": false,
"coreMigrationVersion": "8.8.0",
"typeMigrationVersion": "8.9.0"
}
```
***Adoption***
Several tools provide dashboards-as-code integrations by using the Saved Object API.
- [edctl](https://github.com/elastic/edctl): manages Stack deployments.
- [Elastic Stack terraform provider](https://github.com/elastic/terraform-provider-elasticstack): a dashboard integration is provided through the [Saved object provider](https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/kibana_import_saved_objects)
- [Elastic Agent Integrations](https://docs.elastic.co/en/integrations): Elastic Integrations include ready-made resources such as Dashboards, and imported with Agent into Kibana
***Limitations***
There are a number of limitations with using Saved Objects API.
****General limitations****
- _It is deprecated (since 8.7)_. The existing SO-API is a poor fit for a SAAS-product.
- It is difficult to evolve backward compatibility for all resource types with a single API.
- 1:1 correspondence of storage implementation details (ie. the shape of the SO object, how it persists inside Elasticsearch) and the API layer.
- _No OpenAPI documentation_: The SO-API does not have a standardized documentation to generate client-code.
- _No domain specific validation_: The SO apis do not validate individual content. The payload is a "bag of values", either as string or json.
- _No clear team ownership_: individual teams should own the API layer for their resources
****Dashboard specific limitations****
- _absence of higher level dashboard-specific operation_: e.g. add/remove panels, ...
- _string content requires additional parsing_: this gets specifically tricky when nesting contents (e.g. dashboard panels may have stringified contents too (e.g. maps))
- _string content has very poor readability_: this is important when resources are checked in under version control (e.g. git), and users want to be able to easily identify changes across versions
- sorting keys would reduce visual clutter in diffs (https://github.com/elastic/kibana/issues/167537)
- using an unmapped field would provide better readability and ergonomics (ie indented Json is quite readable) (https://github.com/elastic/kibana/issues/167824)
-----
**Proposal**
Introduce a dedicated REST API to perform Dashboard CRUD, and higher level, domain specific operations.
***Incremental approach***
Given the scope of the work, we would propose an incremental approach of delivery. Below a rough ordering of priority.
****1) Extract basic functionality****
CRUD for dashboards
This is similar to the current SO, support, but
- ensures explicit backward compatibility
- removes stringified json
- adds OpenAPI spec
for example:
```
PUT http://kibanaproject/{spaceid?}/dashboard/{id}
Accept: application/json
Content-Type: application/json
elastic-version: 2024-01-01
{
options: { ...}
panels: [
{ layout: ..., content: ...},
{ layout: ..., content: ...},
]
}
```
****2) Add content validation****
Add content validation for Dashboard contents
****3) Fully type panels****
Fully type API to include all panels as a first level object.
****Scope limits****
To limit scope, consider only:
- by-value panels
- limit full typing to select widget-types: Lens, Maps, Saved Search, and exclude legacy (e.g. agg-based visualizations)
- no versioning of dashboards. Dashboard versioning would still be outside of Kibana (e.g. github)
-----
**Comparisons**
Similar functionality already exists.
***Internal comparisons***
Several other resource types in Kibana already have a dedicated http API. They support basic CRUD, but also domain specific operations.
- [Cases](https://www.elastic.co/guide/en/kibana/current/cases-api.html): beyond CRUD, this also provide higher level operations like CRUD for comments, manipulating individual properties, ...
- [Alerts](https://www.elastic.co/guide/en/kibana/current/alerting-apis.html): beyond CRUD, also provides higher level operations like enable/disable alerts, ...
- [DataViews](https://www.elastic.co/guide/en/kibana/current/data-views-api.html): beyond CRUD, also provides higher level operations like updating individual fields, ...
***External comparisons***
Peer products also provide dedicated http APIs to manage dashboards.
- [DataDog](https://docs.datadoghq.com/api/latest/dashboards/): fully typed aPI for Dasbhoard CRUD, including its individual widgets (~ "panels")
- [Grafana](https://grafana.com/docs/grafana/latest/developers/http_api/dashboard/): this also includes versioning (something considered out-of-scope here)
-----
**Describe a specific use case for the feature:**
Infa-as-code use-cases and machine integrations like Terraform providers, Ansible, ...