Logging not happening on elastic Kubernetes Environment

[Prabakar]

We are using Elastic on Kubernetes (ECK) Enterprise edition, we could not able to see the index creation, search request logs. In general logs will get generated inside /logs/logfile. But log files are not available for ECK environment. We tried to use below properties as well for slow logs. But logging still not happening. Please let us know how to see the logs for all our search, create, update requests. We need log to debug any issues.

PUT //_settings
{
"index.search.slowlog.threshold.query.warn": "1ms",
"index.search.slowlog.threshold.query.info": "1ms",
"index.search.slowlog.threshold.query.debug": "1ms",
"index.search.slowlog.threshold.query.trace": "1ms",
"index.search.slowlog.threshold.fetch.warn": "1ms",
"index.search.slowlog.threshold.fetch.info": "1ms",
"index.search.slowlog.threshold.fetch.debug": "1ms",
"index.search.slowlog.threshold.fetch.trace": "1ms"
}

[x00m]

+1
I need this as well. I am unable to find any logs of the http requests made to Elasticsearch.

@ECK Team, please help!

[legoguy1000 (Alex)]

Have you tried kubectl logs <es pod name> -n <namespace>?

[x00m]

Yes, i even set the logging level to trace. but i am not getting any logs of all the http calls made to Elasticsearch. I can only see logs of incorrect credentials, etc but not of creation of indexes, search calls, etc

[Thibault_Richard (Thibault)]

We are using Elastic on Kubernetes (ECK) Enterprise edition, we could not able to see the index creation, search request logs.

By default Elastisearch doesn't log all requests and this is not specific to ECK.

You can use the Slow logs to log all search and indexing requests by setting the thresholds to 0s:

/_all/_settings -XPUT -d '{
	"index.search.slowlog.level": "trace",
	"index.search.slowlog.threshold.query.trace": "0s",
	"index.indexing.slowlog.level": "trace",
	"index.indexing.slowlog.threshold.index.trace": "0s"
}'

If you want all incoming HTTP requests, the HTTP layer has a tracer logger which can be dynamically activated:

/_cluster/settings -XPUT -d '{
    "transient": {
        "logger.org.elasticsearch.http.HttpTracer": "TRACE"
    }
}'

[Prabakar]

Still I could not able to see the request and response json on logs. I tried by updating below property to debug. But no luck.

"index.search.slowlog.level": "debug",
"index.indexing.slowlog.level": "debug",

[x00m]

Thank you. Is there a way I can specify it in the ECK yaml file under the "config:" section which is under the nodeSets ? What are those setting names?

[Thibault_Richard (Thibault)]

The HttpTracer logger can be configured in the Elasticsearch configuration as described here. Example for ECK:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: foo
spec:
  version: 7.15.2
  nodeSets:
  - name: master
    count: 1
    config:
      node.store.allow_mmap: false
      logger.org.elasticsearch.http.HttpTracer: TRACE

The Slow logs settings are dynamic and are set per-index, which means you cannot set them in the Elasticsearch configuration and should use the update index settings API instead.