Hello,
I have deployed an ECK on GKE following the quickstart documentation and 3 nodes.
The ES YAML file looks like this:
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.6.2
nodeSets:
-name: default
count: 3
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
EOF
And Kibana:
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: quickstart
spec:
version: 7.6.2
count: 1
elasticsearchRef:
name: quickstart
EOF
The GKE has 300 GBs of storage and each node 8 GBs RAM. When I am trying to upload a JSON of size 13 MBs using the Python API it works until the 2.4% and then it complains about:
"TransportError(503, 'cluster_block_exception', 'blocked by: [SERVICE_UNAVAILABLE/2/no master];')"
And when I visit https://ext-kb-IP:5600 I get:
"Kibana server is not ready yet"
kubectl get svc
kubernetes ClusterIP 10.48.0.1 <none> 443/TCP 16m
quickstart-es-default ClusterIP None <none> <none> 12m
quickstart-es-http LoadBalancer 10.48.10.166 ext-IP 9200:31625/TCP 12m
quickstart-es-transport ClusterIP None <none> 9300/TCP 12m
quickstart-kb-http LoadBalancer 10.48.12.76 ext-kb-IP 5601:30168/TCP 9m6s
kubectl get pods
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 1/1 Running 0 111m
quickstart-es-default-1 1/1 Running 0 111m
quickstart-es-default-2 1/1 Running 0 111m
quickstart-kb-655d9b679c-cbp8l 1/1 Running 0 28m
curl -XGET -u "elastic:PASSWORD" -k "https://localhost:9200/_cluster/health?pretty":
{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}
The kibana pod logs:
{"type":"log","@timestamp":"2020-05-12T13:49:28Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"case\" is disabled."}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [37] plugins: [taskManager,siem,licensing,infra,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","taskManager"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","siem"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","infra"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","encryptedSavedObjects"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":6,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","usageCollection"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","metrics"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","canvas"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","apm_oss"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","share"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","home"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","cloud"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","apm"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","graph"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","bfetch"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","savedobjects-service"],"pid":6,"message":"Starting saved objects migrations"}
Could not create APM Agent configuration: Request Timeout after 30000ms
{"type":"log","@timestamp":"2020-05-12T13:50:30Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}
