config:
filebeat:
autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints:
enabled: true
default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
containers:
- name: filebeat
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: default
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
```
Thank you again for your effort in helping!
---
### Author:  [@matschaffer](https://discuss.elastic.co/u/matschaffer)
#### Post date: [August 23, 2022, 7:06am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/6 "2022-08-23T07:06:47Z")
Thanks @papers_hive ! I'm most familiar with using the elasticsearch filebeat module directly and I don't have much experience with hinted k8s configuration.
Do you know if the containers in question are hinted in a way that would activate that module for the elasticsearch container logs?
Do the docs that you found also contain a `elasticsearch.cluster.uuid` field?
It seems like those are the only two that are required by that `getLogs` function to show logs in the UI.
---
### Author:  [@papers\_hive](https://discuss.elastic.co/u/papers_hive)
#### Post date: [August 23, 2022, 7:34am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/7 "2022-08-23T07:34:39Z")
By rerunning and inspecting the before query, I saw that every record has the cluster.uuid in the form:
I do not know exactly how the containers would fetch the logs to elasticsearch. As both `heartbeat` and `metricbeat` were automatically working, I assumed the same for filebeat. And I can see it generates logs files and it is able to see the elasticsearch cluster and the uuid, though somehow these logs are not shipped to the correct place for Kibana to show them.
And I am a bit lost.
---
### Author:  [@matschaffer](https://discuss.elastic.co/u/matschaffer)
#### Post date: [August 24, 2022, 1:26am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/8 "2022-08-24T01:26:37Z")
It's hard to tell from just a screenshot of the doc, but I checked the query on a 8.3 deployment and it looks like it's using the same fields I mentioned above.
If this works, then the UI should be able to show you the same logs as long as you're viewing the same cluster/timerange as the documents.
```auto
POST filebeat-*/_search
{
"fields": [
"@timestamp",
"service.type",
"elasticsearch.cluster.uuid",
"message"
],
"sort": {
"@timestamp": {
"order": "desc"
}
},
"query": {
"bool": {
"filter": [
{
"term": {
"service.type": "elasticsearch"
}
},
{
"exists": {
"field": "elasticsearch.cluster.uuid"
}
}
]
}
},
"_source": false
}
```
The "we are unable to diagnose why" is curious too.
It means this whole section is reaching the end without determining a cause either: [kibana/reason.js at 7.14 · elastic/kibana · GitHub](https://github.com/elastic/kibana/blob/7.14/x-pack/plugins/monitoring/public/components/logs/reason.js#L25) (checks in [kibana/detect\_reason.js at 7.14 · elastic/kibana · GitHub](https://github.com/elastic/kibana/blob/7.14/x-pack/plugins/monitoring/server/lib/logs/detect_reason.js#L11))
I wonder if there might be info the kibana logs that would point to a cause.
---
### Author:  [@papers\_hive](https://discuss.elastic.co/u/papers_hive)
#### Post date: [August 25, 2022, 2:15pm UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/9 "2022-08-25T14:15:39Z")
Hi! Sorry for the delayed reply. Got a bit behind in other stuff.
This is the reply when running that query on the cluster:
---
### Author:  [@matschaffer](https://discuss.elastic.co/u/matschaffer)
#### Post date: [August 26, 2022, 2:12am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/10 "2022-08-26T02:12:49Z")
No problem and thanks for the response! That sure looks like it should match the query.
And you're definitely viewing the same cluster uuid shown in `elastic.cluster.uuid` right?
Have you checked your kibana logs or browser console for any errors?
I wonder if there might be some permissions problem causing trouble. Technically the query run by the UI is probably `POST *:filebeat-*,filebeat-*/_search`, so maybe try that to see if there might be some CCS execution problem as well.
---
### Author:  [@papers\_hive](https://discuss.elastic.co/u/papers_hive)
#### Post date: [August 29, 2022, 6:58am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/11 "2022-08-29T06:58:17Z")
Yes, we are viewing the same cluster uuid shown in `elastic.cluster.uuid`.
Where should I run that query? On the Kibana dashboard?
---
### Author:  [@matschaffer](https://discuss.elastic.co/u/matschaffer)
#### Post date: [August 31, 2022, 1:11am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/12 "2022-08-31T01:11:37Z")
Same place you did the last one. I use kibana's dev tools usually.
---
### Author:  [@papers\_hive](https://discuss.elastic.co/u/papers_hive)
#### Post date: [August 31, 2022, 7:43am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/13 "2022-08-31T07:43:56Z")
I run that query on Dev Tools and here a screen shot of the reply:
Kind of as usual, it sees the index and there are \>10k items in it. All with an uuid.
---
### Author:  [@matschaffer](https://discuss.elastic.co/u/matschaffer)
#### Post date: [September 5, 2022, 5:30am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/14 "2022-09-05T05:30:38Z")
Yeah, that looks reasonable too. If you can upgrade to at least 7.15, you could try setting `monitoring.ui.debug_mode: true` to see what queries the UI is executing exactly.
Or on 7.17 you could enable APM traces as shown on [3 tips to identify Kibana optimizing potential | Elastic Blog](https://www.elastic.co/blog/3-tips-to-identify-kibana-optimizing-potential) - that would capture the ES queries into an APM trace.
It's also possible we're looking at a 7.14 bug here, but I can't recall anything like this offhand.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 3, 2022, 5:31am UTC](https://discuss.elastic.co/t/kibana-states-no-log-data-found-on-eck/312064/15 "2022-10-03T05:31:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.