I restarted it and everything is ok now.
I guess based on my settings i needed a minimum of 3 nodes to be available for a master to to be elected and we didn't have 3 nodes at the time.
This was for Elasticsearch 5.4
We have a 5 node cluster which was running fine until last week. This week, even though processes seem to be running they are either not logging or logging errors like:
[o.e.a.a.i.c.TransportCreateIndexAction] [CO4SCH010031328] no known master node, scheduling a retry
My understanding is ES is by default in HA mode and I don't need to set anything specifically for that to happen other than these important settings:
discovery.zen.ping.unicast.hosts: ["server1", "server2", "server3", "server4", "server5"]