Filebeat not logging to elasticsearch

[whoatemyjam (shantanud)]

Hi
Please if someone can help with the issue will be highly appreciated
we have openshift cluster 3.11 and trying to replace fluentd with filebeat.
we have deployed filebeat 6.0 which is compatible with current es on clutser 5.6.2 and the disabled the fluentd pods but cannot find anything logged in .
Fairly new to the stuff and tried to search but nothing relevant came up.
Anyone got any suggestions or would require any specific logs please.

from the filebeat pod , logs in /var/lib/docker/containers/
the *-json.log is empty

Thanks and regards

[whoatemyjam (shantanud)]

any one ?

[kumarabhi (Abhishek)]

Please provide the filebeat configuration file (default :- filebeat.yml ). How do you start Fileebeat ?
Did you check the filebeat logs ?

https://www.elastic.co/guide/en/beats/filebeat/master/directory-layout.html

[whoatemyjam (shantanud)]

hi Abhi
Thanks for looking into it
Please see below the requested files

> filbeat.yml
> > filebeat.config:
> >   inputs:
> >     # Mounted `filebeat-inputs` configmap:
> >     path: ${path.config}/inputs.d/*.yml
> >     # Reload inputs configs as they change:
> >     reload.enabled: false
> >   modules:
> >     path: ${path.config}/modules.d/*.yml
> >     # Reload module configs as they change:
> >     reload.enabled: false
> > 
> > # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
> > #filebeat.autodiscover:
> > #  providers:
> > #    - type: kubernetes
> > #      hints.enabled: true
> > 
> > processors:
> >   - add_cloud_metadata:
> > 
> > cloud.id: ${ELASTIC_CLOUD_ID}
> > cloud.auth: ${ELASTIC_CLOUD_AUTH}
> > 
> > output.elasticsearch:
> >    hosts: '${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}'
> >    username: '${ELASTICSEARCH_USERNAME:}'
> >    password: '${ELASTICSEARCH_PASSWORD:}

and logs from filebeat

TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x20cf47e, Device:0xfd02}}}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}", "input":common.MapStr{"type":"docker"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc42082be10), Source:"/var/lib/docker/containers/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03-json.log", Offset:3964557, Timestamp:time.Time{wall:0xbf64d335b1d64fff, ext:71633082111848, loc:(*time.Location)(0x21f36a0)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x20cf47e, Device:0xfd02}}}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2019-10-25T10:58:32.035Z WARN elasticsearch/client.go:539 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x3e14f8a, ext:63707597902, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"source":"/var/lib/docker/containers/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03-json.log", "log":common.MapStr{"file":common.MapStr{"path":"/var/lib/docker/containers/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03-json.log"}}, "prospector":common.MapStr{"type":"docker"}, "input":common.MapStr{"type":"docker"}, "kubernetes":common.MapStr{"labels":common.MapStr{"controller-revision-hash":"1918509931", "k8s-app":"filebeat", "pod-template-generation":"1"}, "pod":common.MapStr{"uid":"9705e669-f66f-11e9-ba52-566f75280038", "name":"filebeat-daemonset-2vwf4"}, "node":common.MapStr{"name":""}, "container":common.MapStr{"name":"filebeat"}, "namespace":"filebeat"}, "beat":common.MapStr{"version":"6.8.3", "name":"filebeat-daemonset-2vwf4", "hostname":"filebeat-daemonset-2vwf4"}, "host":common.MapStr{"name":"filebeat-daemonset-2vwf4"}, "message":"2019-10-25T10:58:22.055Z\tWARN\telasticsearch/client.go:539\tCannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x38d46b3f, ext:63707597891, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"prospector":common.MapStr{"type":"docker"}, "source":"/var/lib/docker/containers/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03-json.log", "offset":4712836, "stream":"stderr", "kubernetes":common.MapStr{"namespace":"filebeat", "labels":common.MapStr{"controller-revision-hash":"1918509931", "k8s-app":"filebeat", "pod-template-generation":"1"}, "pod":common.MapStr{"uid":"9705e669-f66f-11e9-ba52-566f75280038", "name":"filebeat-daemonset-2vwf4"}, "node":common.MapStr{"name":"g"}, "container":common.MapStr{"name":"filebeat"}}, "beat":common.MapStr{"hostname":"filebeat-daemonset-2vwf4", "version":"6.8.3", "name":"filebeat-daemonset-2vwf4"}, "host":common.MapStr{"name":"filebeat-daemonset-2vwf4"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/lib/docker/containers/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03/219bcc9d0cebe17f0031bc6a3a966b7e7c02d4abc689ba465f505feae35dfb03-json.log"}}, "message":"2019-10-25T10:58:11.953Z\tINFO\tlog/harvester.go:268\tFile was truncated. Begin reading file from offset 0:

could the error

[FORBIDDEN/12/index read-only / allow delete (api)];"}

be something to look at , its same error on elasticsearch pods that i see

Thanks and regards

[whoatemyjam (shantanud)]

so i checked for possible solutions and i think the disk space is fine on elastic search pods
elastic search pod -

1. 10G 6.2G 3.9G 62% /usr/share/elasticsearch/data
2. 8.9G 1.2G 89% /usr/share/elasticsearch/data
   10G 9.1G 985M 91% /usr/share/elasticsearch/data

another was to put settings but i am not exactly sure how to do that

Thanks