Partial json results when nodes /var partitions are filled

We observed a situation recently where a large increase in the amount of
data being ingested into ES caused log files in /var to fill to capacity on
each node in a six node cluster.

Attempts to curl to a master VIP front-ended on the cluster would return a
portion of the full json, but not all of it. Imagine expecting ten "pages"
of some type of information and only receiving 2-3 when it suddenly stops.

I first saw this when manually using curl on a Linux client. I mistakenly
thought it might be a client issue or a load balancer or proxy issue
between me and the the cluster.

The next symptom came in the form of Elastic HQ being able to connect to
the cluster but producing "Error: Unable to Read Node List".

The third symptom came from Kibana being unable to pull up any dashboards,
timing out and proclaiming that our ES cluster was down.

Monitoring should have been enabled to monitor /var and we hadn't received
any alerts on it, so of course we didn't think to stop and check any of the
filesystems for out-of-space issues. After we noticed this and corrected
the problem, all our problems went away.

The question is: Why would /var filling cause the cluster to return
partial json in this manner?

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/45780f2b-df8d-4780-a107-bd8c6a58d325%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.