Persistent Kibana shard failure error for oldest shown data

[cealliance (Connor)]

When opening any visualization, dashboard, or the discover tab, an error appears explaining that ~%80 of shards failed. Unless the time span is set to very small increments, e.g. 15 minutes, the oldest %80 of data never appears in the search regardless of filters.

The kibana log output shows nothing, but the UI error reports the following:

illegal_argument_exception

date[+51090849-12-08T18:11:39.201Z] is after 2262-04-11T23:47:16.854775807 and cannot be stored in nanosecond resolution

All dates have been verified to be normal dates around 2020-2021. Plus, all data can be successfully shown without errors, but only when requested in 4 hour segments max.

I am at a bit of a loss here. What are the next steps I should take to diagnose this issue?

[warkolm (Mark Walkom)]

Welcome to our community!

What is the health of your Elasticsearch cluster?

[cealliance (Connor)]

Thank you for the warm welcome and the fast reply!

Unfortunately the cluster health is green.

{
  "cluster_name" : "nc-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 279,
  "active_shards" : 279,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[cealliance (Connor)]

After this step, what should I do to troubleshoot next?