Exception while openig kibana filebeat dashboard

i and newbie to Elasticsearh and have configured elasticsearch + kibana on a windows server and it was perfectly working fine for last few days.

  1. Today when i tried to open kibana dashboard for winlogbeat and filebeat, it send an exception about something disk usage (although there was around 4GB available). So i created some more space in C drive and tried again. After a while winlogbeat loads successfully but on filebeat dashboard it is throwing exception.

"search_phase_execution_exception
all shards failed

Error: Service Unavailable
at Fetch._callee3$ (http://10.10.100.88:5601/33813/bundles/core/core.entry.js:34:109213)
at l (http://10.10.100.88:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155323)
at Generator._invoke (http://10.10.100.88:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155076)
at Generator.forEach.e. [as next] (http://10.10.100.88:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155680)
at fetch_asyncGeneratorStep (http://10.10.100.88:5601/33813/bundles/core/core.entry.js:34:102354)
at _next (http://10.10.100.88:5601/33813/bundles/core/core.entry.js:34:102670)"

please advise

  1. and also can i delete data from this folder as it is consuming lot of space:
    C:\ProgramData\Elastic\Elasticsearch\data\nodes\0\indices\or209yvfTsKkTPg5u_1QRw\0\index

Update= i have deleted the filebeat-* index pattern and tried to reinstall filebeat. when i run the PS cmd .\filebeat.exe setup -e. it shows an error..
ERROR instance/beat.go:951 Exiting: 1 error: fileset iis/var is configured but doesn't exist
Exiting: 1 error: fileset iis/var is configured but doesn't exist

can some body suggest how to solve this?

What do your Elasticsearch logs show?

Always use the API to delete data from Elasticsearch, never touch directories/folders directly on the filesystem as this can have adverse impacts.

thanks here is the elasticsearch log:

[2020-09-07T01:30:00,003][INFO ][o.e.x.m.MlDailyMaintenanceService] [MGMT] triggering scheduled [ML] maintenance tasks
[2020-09-07T01:30:00,050][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [MGMT] Deleting expired data
[2020-09-07T01:30:00,112][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [MGMT] Completed deletion of expired ML data
[2020-09-07T01:30:00,128][INFO ][o.e.x.m.MlDailyMaintenanceService] [MGMT] Successfully completed [ML] maintenance tasks
[2020-09-07T08:33:18,920][INFO ][o.e.c.m.MetadataIndexTemplateService] [MGMT] adding template [filebeat-7.9.1] for index patterns [filebeat-7.9.1-]
[2020-09-07T12:00:02,391][INFO ][o.e.c.m.MetadataCreateIndexService] [MGMT] [.monitoring-es-7-2020.09.07] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0]
[2020-09-07T12:00:09,561][INFO ][o.e.c.m.MetadataCreateIndexService] [MGMT] [.monitoring-kibana-7-2020.09.07] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0]
[2020-09-07T12:00:35,887][INFO ][o.e.c.m.MetadataCreateIndexService] [MGMT] [.watcher-history-11-2020.09.07] creating index, cause [auto(bulk api)], templates [.watch-history-11], shards [1]/[0]
[2020-09-07T12:00:35,934][INFO ][o.e.x.i.IndexLifecycleTransition] [MGMT] moving index [.watcher-history-11-2020.09.07] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [watch-history-ilm-policy]
[2020-09-07T12:00:36,053][INFO ][o.e.c.m.MetadataMappingService] [MGMT] [.watcher-history-11-2020.09.07/JMJSbFcbQhecP4prF-7fGg] update_mapping [_doc]
[2020-09-07T12:00:36,926][INFO ][o.e.c.m.MetadataMappingService] [MGMT] [.watcher-history-11-2020.09.07/JMJSbFcbQhecP4prF-7fGg] update_mapping [_doc]
[2020-09-07T12:00:37,004][INFO ][o.e.c.m.MetadataMappingService] [MGMT] [.watcher-history-11-2020.09.07/JMJSbFcbQhecP4prF-7fGg] update_mapping [_doc]
[2020-09-07T13:30:00,018][INFO ][o.e.x.s.SnapshotRetentionTask] [MGMT] starting SLM retention snapshot cleanup task
[2020-09-07T13:30:00,033][INFO ][o.e.x.s.SnapshotRetentionTask] [MGMT] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2020-09-07T13:46:49,813][INFO ][o.e.c.m.MetadataIndexTemplateService] [MGMT] adding template [filebeat-7.9.1] for index patterns [filebeat-7.9.1-
]
[2020-09-07T18:01:52,349][INFO ][o.e.c.m.MetadataIndexTemplateService] [MGMT] adding template [filebeat-7.9.1] for index patterns [filebeat-7.9.1-*]

There's not much useful there unfortunately.

Is there anything in the Kibana logs? What is the current state of the cluster, you can check this this way?

GET _cat/health?v

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1599461908 06:58:28 elasticsearch yellow 1 1 23 23 0 0 2 0 - 92.0%

GET _cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-kibana-7-2020.09.07 hjWSiJxzRy2cWZmcrIW3gw 1 0 4928 0 890.4kb 890.4kb
green open .triggered_watches wcxrRSwaTfek1V-qYNyTeg 1 0 0 0 543.5kb 543.5kb
green open .monitoring-kibana-7-2020.09.06 DKjiZhtNQfuqCyDaFTY-6Q 1 0 10380 0 1.7mb 1.7mb
green open .kibana-event-log-7.9.0-000001 Cpkg_yseQGmnkQFoaYnCIA 1 0 8 0 37.7kb 37.7kb
yellow open winlogbeat-7.9.1-2020.09.03-000001 KQb6duiARbqIroHSuti3qg 1 1 176184 0 112.3mb 112.3mb
green open .apm-agent-configuration wOBHCR0ATH2_9dBjY9OuDA 1 0 0 0 208b 208b
green open .kibana_1 06rMMN6tQs-Rab5dCznf6w 1 0 2143 344 11.4mb 11.4mb
green open .ml-config I6rJgG4GQi-Wvn8uH88Vuw 1 0 20 0 102.7kb 102.7kb
green open .security-7 BqAs1_ymSom-Qe90kk9V7w 1 0 54 0 153.9kb 153.9kb
green open .apm-custom-link JO0HixI_QReYIbxu9AALaQ 1 0 0 0 208b 208b
yellow open filebeat-7.9.1-2020.09.06-000001 r4e7VlwoRRuTCyAURGdu1w 1 1 0 0 208b 208b
green open .kibana_task_manager_1 cH0jcwgUQh2zrpHKj32z0Q 1 0 6 3825 776.6kb 776.6kb
green open .monitoring-es-7-2020.09.06 rjGaf99gSOSZX6Rt2RxHVQ 1 0 160969 113589 74.4mb 74.4mb
green open .monitoring-es-7-2020.09.07 6Yx8PnotREWtoN-vb9RmxA 1 0 81464 61498 38mb 38mb
green open .monitoring-alerts-7 0JHshLpXToa2m-LaEWKNDQ 1 0 1 1275 106.7kb 106.7kb
green open .async-search vc_x3pXaRgSddD-DPyHquA 1 0 0 0 3.3kb 3.3kb
green open .watches LQ6Z90NWQBammkPr5XrZLQ 1 0 6 7656 8.2mb 8.2mb

Kibana log:

2020-09-06T08:15:20.152+1200 INFO instance/beat.go:640 Home path: [C:\Program Files\filebeat] Config path: [C:\Program Files\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2020-09-06T08:15:20.163+1200 INFO instance/beat.go:648 Beat ID: cbea1ece-1b9f-4c81-806b-bb192f74e798
2020-09-06T08:15:20.182+1200 INFO [beat] instance/beat.go:976 Beat info {"system_info": {"beat": {"path": {"config": "C:\Program Files\filebeat", "data": "C:\ProgramData\filebeat", "home": "C:\Program Files\filebeat", "logs": "C:\ProgramData\filebeat\logs"}, "type": "filebeat", "uuid": "cbea1ece-1b9f-4c81-806b-bb192f74e798"}}}
2020-09-06T08:15:20.183+1200 INFO [beat] instance/beat.go:985 Build info {"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T19:58:48.000Z", "version": "7.9.1"}}}
2020-09-06T08:15:20.184+1200 INFO [beat] instance/beat.go:988 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2020-09-06T08:15:20.189+1200 INFO [beat] instance/beat.go:992 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-08-21T23:23:19.87+12:00","name":"AP08","ip":["10.10.100.102/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.1282 (WinBuild.160101.0800)","mac":["00:50:56:b9:30:e4"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.1282"},"timezone":"NZST","timezone_offset_sec":43200,"id":"8470a48d-de02-447e-99c1-011d699ae34e"}}}
2020-09-06T08:15:20.192+1200 INFO [beat] instance/beat.go:1021 Process info {"system_info": {"process": {"cwd": "C:\Windows\system32", "exe": "C:\Program Files\filebeat\filebeat.exe", "name": "filebeat.exe", "pid": 38728, "ppid": 692, "start_time": "2020-09-06T08:15:18.468+1200"}}}
2020-09-06T08:15:20.192+1200 INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.9.1
2020-09-06T08:15:20.194+1200 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.9.1' as ILM is enabled.
2020-09-06T08:15:20.201+1200 INFO eslegclient/connection.go:99 elasticsearch url: http://10.10.100.88:9200
2020-09-06T08:15:20.215+1200 INFO [publisher] pipeline/module.go:113 Beat name: AP08
2020-09-06T08:15:20.268+1200 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-09-06T08:15:20.269+1200 INFO instance/beat.go:450 filebeat start running.
2020-09-06T08:15:20.377+1200 INFO memlog/store.go:119 Loading data file of 'C:\ProgramData\filebeat\registry\filebeat' succeeded. Active transaction id=1933745420
2020-09-06T08:15:20.451+1200 INFO memlog/store.go:124 Finished loading transaction log file for 'C:\ProgramData\filebeat\registry\filebeat'. Active transaction id=1933749559
2020-09-06T08:15:20.500+1200 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 5381
2020-09-06T08:15:20.501+1200 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2020-09-06T08:15:20.513+1200 INFO beater/crawler.go:148 Stopping Crawler
2020-09-06T08:15:20.513+1200 INFO beater/crawler.go:158 Stopping 0 inputs
2020-09-06T08:15:20.513+1200 INFO beater/crawler.go:178 Crawler stopped
2020-09-06T08:15:20.513+1200 INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2020-09-06T08:15:20.513+1200 INFO [registrar] registrar/registrar.go:166 Ending Registrar
2020-09-06T08:15:20.623+1200 INFO [registrar] registrar/registrar.go:137 Registrar stopped
2020-09-06T08:15:20.655+1200 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":218,"time":{"ms":218}},"total":{"ticks":608,"time":{"ms":608},"value":608},"user":{"ticks":390,"time":{"ms":390}}},"handles":{"open":231},"info":{"ephemeral_id":"a3c45770-924a-4e38-a131-910ea16909bd","uptime":{"ms":1505}},"memstats":{"gc_next":35015200,"memory_alloc":18731448,"memory_total":88252664,"rss":72908800},"runtime":{"goroutines":31}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4}}}}}
2020-09-06T08:15:20.655+1200 INFO [monitoring] log/log.go:154 Uptime: 1.5072252s
2020-09-06T08:15:20.655+1200 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-09-06T08:15:20.656+1200 INFO instance/beat.go:456 filebeat stopped.
2020-09-06T08:15:20.658+1200 ERROR instance/beat.go:951 Exiting: Failed to start crawler: creating module reloader failed: fileset iis/var is configured but doesn't exist