Dashboard panels don't show data though underlying visualizations appear to work

Some panels on a dashboard no longer show data though when you go in to edit the visualization, it does show the data.

The affected visualizations are based on follower indexes that had stopped receiving data from the leader due to an error similar to this:

"exception": { "type": "resource_not_found_exception", "reason": "Operations are no longer available for replicating. Existing retention leases [[RetentionLease{id='peer_recovery/juwVUvDtSNWMWVFKf93Kog', retainingSequenceNumber=148206781, timestamp=1658957145194, source='peer recovery'}, RetentionLease{id='peer_recovery/-B_82BCdTamNz3nciFbulg', retainingSequenceNumber=148206781, timestamp=1658957145194, source='peer recovery'}, RetentionLease{id='peer_recovery/VY1CY7peSbqCreHFbL-xcA', retainingSequenceNumber=148206781, timestamp=1658957145194, source='peer recovery'}]]; maybe increase the retention lease period setting [index.soft_deletes.retention_lease.period]?", "requested_operations_missing": [ "147970877", "147975996" ], "caused_by": { "type": "illegal_state_exception", "reason": "Not all operations between from_seqno [147970877] and to_seqno [147975996] found; expected seqno [147970877]; found [Index{id='hraja1', type='_doc', seqNo=147970971, primaryTerm=6, version=215, autoGeneratedIdTimestamp=-1}]"

The follower indexes have since been recreated to resolve the error and we are now these indexes are getting current data from the leader and we can see it in the visualization itself but still no data is showing on the dashboard panel. Another note: for the panels based on TVSB visualizations, I do have to click from the Data tab on the visualization to the Panel Options tab for the data to appear. For the one panel that is based on a Lens, the data shows up right away when I go to Edit Lens.

I have tried:

  • Going to Edit Visualization, clicking on the Panel Options so the data appears, then doing a Save and Return, but the dashboard panel still does not show any data.

  • Saving one of the visualizations to the library, deleting the panel from the dashboard, saving the dashboard, then adding the visualization from the library and still no data shows up on the dashboard panel.

I think the data stopped showing on the panels when the follower indexes first broke, but I can't be sure.

We are on v7.17.3