I have a K8s ECK deployed as this :
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: dev
namespace: monitoring
spec:
version: 8.8.0
nodeSets:
- name: dev
count: 3
podTemplate:
spec:
containers:
- name: elasticsearch
readinessProbe:
exec:
command:
- bash
- -c
- /mnt/elastic-internal/scripts/readiness-probe-script.sh
failureThreshold: 3
initialDelaySeconds: 15
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 20
env:
- name: READINESS_PROBE_TIMEOUT
value: "20"
resources:
requests:
memory: 16Gi
cpu: 5
limits:
memory: 20Gi
cpu: 10
config:
node.roles: ["master", "data_hot", "data_content", "ingest", "ml", "transform", "remote_cluster_client"]logger.level: "ERROR"
xpack.monitoring.collection.enabled: true
volumeClaimTemplates:- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
storageClassName: "local-path"
accessModes:- ReadWriteOnce
resources:
requests:
storage: 100Gi
- ReadWriteOnce
- metadata:
but during usage and testing with metricbeats / filebeats / other stuff i'm getting continuously spammed in the elasticsearch log by these messages :
"@timestamp":"2023-05-29T10:11:47.606Z", "log.level": "WARN", "message":"Failed to get search application count to include in Enterprise Search usage", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[dev-es-dev-2][management][T#1]","log.logger":"org.elasticsearch.xpack.application.EnterpriseSearchUsageTransportAction","elasticsearch.cluster.uuid":"KyO9qI_GRZCsSRUUMOvVDw","elasticsearch.node.id":"VWClbmvDQR-gjAtUnNmS9g","elasticsearch.node.name":"dev-es-dev-2","elasticsearch.cluster.name":"dev","error.type":"java.util.concurrent.ExecutionException","error.message":"org.elasticsearch.ElasticsearchSecurityException: Current license is non-compliant for search application and behavioral analytics. Current license is active basic license. Search Applications and behavioral analytics require an active trial, platinum or enterprise license.","error.stack_trace":"java.util.concurrent.ExecutionException: org.elasticsearch.ElasticsearchSecurityException: Current license is non-compliant for search application and behavioral analytics. Current license is active basic license. Search Applications and behavioral analytics require an active trial, platinum or enterprise license.\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.PlainActionFuture$Sync.getValue(PlainActionFuture.java:340)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:327)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:95)\n\tat org.elasticsearch.application@8.8.0/org.elasticsearch.xpack.application.EnterpriseSearchUsageTransportAction.masterOperation(EnterpriseSearchUsageTransportAction.java:92)\n\tat org.elasticsearch.application@8.8.0/org.elasticsearch.xpack.application.EnterpriseSearchUsageTransportAction.masterOperation(EnterpriseSearchUsageTransportAction.java:40)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:124)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:235)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: org.elasticsearch.ElasticsearchSecurityException: Current license is non-compliant for search application and behavioral analytics. Current license is active basic license. Search Applications and behavioral analytics require an active trial, platinum or enterprise license.\n\tat org.elasticsearch.application@8.8.0/org.elasticsearch.xpack.application.utils.LicenseUtils.newComplianceException(LicenseUtils.java:33)\n\tat org.elasticsearch.application@8.8.0/org.elasticsearch.xpack.application.utils.LicenseUtils.runIfSupportedLicense(LicenseUtils.java:46)\n\tat org.elasticsearch.application@8.8.0/org.elasticsearch.xpack.application.search.action.SearchApplicationTransportAction.doExecute(SearchApplicationTransportAction.java:55)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:151)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:440)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:1003)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:969)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$7(AuthorizationService.java:454)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:158)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:180)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:444)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:420)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$3(AuthorizationService.java:321)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:158)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$0(RBACEngine.java:146)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:158)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$1(CompositeRolesStore.java:202)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:158)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRole(CompositeRolesStore.java:210)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:192)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:143)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:323)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:158)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:93)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:257)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:167)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:110)\n\tat org.elasticsearch.xcore@8.8.0/org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:165)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.authz.AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(AuthorizationUtils.java:146)\n\tat org.elasticsearch.security@8.8.0/org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:106)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:199)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:57)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.OriginSettingClient.doExecute(OriginSettingClient.java:43)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:379)\n\tat org.elasticsearch.server@8.8.0/org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:365)\n\tat org.elasticsearch.application@8.8.0/org.elasticsearch.xpack.application.EnterpriseSearchUsageTransportAction.masterOperation(EnterpriseSearchUsageTransportAction.java:89)\n\t... 9 more\n"}
to the point the log rate overwhelms the system :
"...to create fsnotify watcher: too many open files..."
The solution so far was to set the logging to "ERROR" but i might miss important warnings.
I do not need a license, i just need to use the free features and make some sense of the logs ( and not have like 100Gb of logs per day just with useless messages ).
I could not find the documentation on this, so i'm turning to your advice.
Thanks !