Exception while openig kibana filebeat dashboard

Elastic Stack Kibana
1

i and newbie to Elasticsearh and have configured elasticsearch + kibana on a windows server and it was perfectly working fine for last few days.

  1. Today when i tried to open kibana dashboard for winlogbeat and filebeat, it send an exception about something disk usage (although there was around 4GB available). So i created some more space in C drive and tried again. After a while winlogbeat loads successfully but on filebeat dashboard it is throwing exception.

"search_phase_execution_exception
all shards failed

Error: Service Unavailable
at Fetch._callee3$ (http://10.10.100.88:5601/33813/bundles/core/core.entry.js:34:109213)
at l (http://10.10.100.88:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155323)
at Generator._invoke (http://10.10.100.88:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155076)
at Generator.forEach.e. [as next] (http://10.10.100.88:5601/33813/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155680)
at fetch_asyncGeneratorStep (http://10.10.100.88:5601/33813/bundles/core/core.entry.js:34:102354)
at _next (http://10.10.100.88:5601/33813/bundles/core/core.entry.js:34:102670)"

please advise

  1. and also can i delete data from this folder as it is consuming lot of space:
    C:\ProgramData\Elastic\Elasticsearch\data\nodes\0\indices\or209yvfTsKkTPg5u_1QRw\0\index
2

Update= i have deleted the filebeat-* index pattern and tried to reinstall filebeat. when i run the PS cmd .\filebeat.exe setup -e. it shows an error..
ERROR instance/beat.go:951 Exiting: 1 error: fileset iis/var is configured but doesn't exist
Exiting: 1 error: fileset iis/var is configured but doesn't exist

can some body suggest how to solve this?

3

What do your Elasticsearch logs show?

Always use the API to delete data from Elasticsearch, never touch directories/folders directly on the filesystem as this can have adverse impacts.

4

thanks here is the elasticsearch log:

[2020-09-07T01:30:00,003][INFO ][o.e.x.m.MlDailyMaintenanceService] [MGMT] triggering scheduled [ML] maintenance tasks
[2020-09-07T01:30:00,050][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [MGMT] Deleting expired data
[2020-09-07T01:30:00,112][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [MGMT] Completed deletion of expired ML data
[2020-09-07T01:30:00,128][INFO ][o.e.x.m.MlDailyMaintenanceService] [MGMT] Successfully completed [ML] maintenance tasks
[2020-09-07T08:33:18,920][INFO ][o.e.c.m.MetadataIndexTemplateService] [MGMT] adding template [filebeat-7.9.1] for index patterns [filebeat-7.9.1-]
[2020-09-07T12:00:02,391][INFO ][o.e.c.m.MetadataCreateIndexService] [MGMT] [.monitoring-es-7-2020.09.07] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0]
[2020-09-07T12:00:09,561][INFO ][o.e.c.m.MetadataCreateIndexService] [MGMT] [.monitoring-kibana-7-2020.09.07] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0]
[2020-09-07T12:00:35,887][INFO ][o.e.c.m.MetadataCreateIndexService] [MGMT] [.watcher-history-11-2020.09.07] creating index, cause [auto(bulk api)], templates [.watch-history-11], shards [1]/[0]
[2020-09-07T12:00:35,934][INFO ][o.e.x.i.IndexLifecycleTransition] [MGMT] moving index [.watcher-history-11-2020.09.07] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [watch-history-ilm-policy]
[2020-09-07T12:00:36,053][INFO ][o.e.c.m.MetadataMappingService] [MGMT] [.watcher-history-11-2020.09.07/JMJSbFcbQhecP4prF-7fGg] update_mapping [_doc]
[2020-09-07T12:00:36,926][INFO ][o.e.c.m.MetadataMappingService] [MGMT] [.watcher-history-11-2020.09.07/JMJSbFcbQhecP4prF-7fGg] update_mapping [_doc]
[2020-09-07T12:00:37,004][INFO ][o.e.c.m.MetadataMappingService] [MGMT] [.watcher-history-11-2020.09.07/JMJSbFcbQhecP4prF-7fGg] update_mapping [_doc]
[2020-09-07T13:30:00,018][INFO ][o.e.x.s.SnapshotRetentionTask] [MGMT] starting SLM retention snapshot cleanup task
[2020-09-07T13:30:00,033][INFO ][o.e.x.s.SnapshotRetentionTask] [MGMT] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2020-09-07T13:46:49,813][INFO ][o.e.c.m.MetadataIndexTemplateService] [MGMT] adding template [filebeat-7.9.1] for index patterns [filebeat-7.9.1-]
[2020-09-07T18:01:52,349][INFO ][o.e.c.m.MetadataIndexTemplateService] [MGMT] adding template [filebeat-7.9.1] for index patterns [filebeat-7.9.1-*]

5

There's not much useful there unfortunately.

Is there anything in the Kibana logs? What is the current state of the cluster, you can check this this way?

6

GET _cat/health?v

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1599461908 06:58:28 elasticsearch yellow 1 1 23 23 0 0 2 0 - 92.0%

GET _cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-kibana-7-2020.09.07 hjWSiJxzRy2cWZmcrIW3gw 1 0 4928 0 890.4kb 890.4kb
green open .triggered_watches wcxrRSwaTfek1V-qYNyTeg 1 0 0 0 543.5kb 543.5kb
green open .monitoring-kibana-7-2020.09.06 DKjiZhtNQfuqCyDaFTY-6Q 1 0 10380 0 1.7mb 1.7mb
green open .kibana-event-log-7.9.0-000001 Cpkg_yseQGmnkQFoaYnCIA 1 0 8 0 37.7kb 37.7kb
yellow open winlogbeat-7.9.1-2020.09.03-000001 KQb6duiARbqIroHSuti3qg 1 1 176184 0 112.3mb 112.3mb
green open .apm-agent-configuration wOBHCR0ATH2_9dBjY9OuDA 1 0 0 0 208b 208b
green open .kibana_1 06rMMN6tQs-Rab5dCznf6w 1 0 2143 344 11.4mb 11.4mb
green open .ml-config I6rJgG4GQi-Wvn8uH88Vuw 1 0 20 0 102.7kb 102.7kb
green open .security-7 BqAs1_ymSom-Qe90kk9V7w 1 0 54 0 153.9kb 153.9kb
green open .apm-custom-link JO0HixI_QReYIbxu9AALaQ 1 0 0 0 208b 208b
yellow open filebeat-7.9.1-2020.09.06-000001 r4e7VlwoRRuTCyAURGdu1w 1 1 0 0 208b 208b
green open .kibana_task_manager_1 cH0jcwgUQh2zrpHKj32z0Q 1 0 6 3825 776.6kb 776.6kb
green open .monitoring-es-7-2020.09.06 rjGaf99gSOSZX6Rt2RxHVQ 1 0 160969 113589 74.4mb 74.4mb
green open .monitoring-es-7-2020.09.07 6Yx8PnotREWtoN-vb9RmxA 1 0 81464 61498 38mb 38mb
green open .monitoring-alerts-7 0JHshLpXToa2m-LaEWKNDQ 1 0 1 1275 106.7kb 106.7kb
green open .async-search vc_x3pXaRgSddD-DPyHquA 1 0 0 0 3.3kb 3.3kb
green open .watches LQ6Z90NWQBammkPr5XrZLQ 1 0 6 7656 8.2mb 8.2mb

Kibana log:

2020-09-06T08:15:20.152+1200 INFO instance/beat.go:640 Home path: [C:\Program Files\filebeat] Config path: [C:\Program Files\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2020-09-06T08:15:20.163+1200 INFO instance/beat.go:648 Beat ID: cbea1ece-1b9f-4c81-806b-bb192f74e798
2020-09-06T08:15:20.182+1200 INFO [beat] instance/beat.go:976 Beat info {"system_info": {"beat": {"path": {"config": "C:\Program Files\filebeat", "data": "C:\ProgramData\filebeat", "home": "C:\Program Files\filebeat", "logs": "C:\ProgramData\filebeat\logs"}, "type": "filebeat", "uuid": "cbea1ece-1b9f-4c81-806b-bb192f74e798"}}}
2020-09-06T08:15:20.183+1200 INFO [beat] instance/beat.go:985 Build info {"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T19:58:48.000Z", "version": "7.9.1"}}}
2020-09-06T08:15:20.184+1200 INFO [beat] instance/beat.go:988 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2020-09-06T08:15:20.189+1200 INFO [beat] instance/beat.go:992 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-08-21T23:23:19.87+12:00","name":"AP08","ip":["10.10.100.102/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.1282 (WinBuild.160101.0800)","mac":["00:50:56:b9:30:e4"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.1282"},"timezone":"NZST","timezone_offset_sec":43200,"id":"8470a48d-de02-447e-99c1-011d699ae34e"}}}
2020-09-06T08:15:20.192+1200 INFO [beat] instance/beat.go:1021 Process info {"system_info": {"process": {"cwd": "C:\Windows\system32", "exe": "C:\Program Files\filebeat\filebeat.exe", "name": "filebeat.exe", "pid": 38728, "ppid": 692, "start_time": "2020-09-06T08:15:18.468+1200"}}}
2020-09-06T08:15:20.192+1200 INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.9.1
2020-09-06T08:15:20.194+1200 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.9.1' as ILM is enabled.
2020-09-06T08:15:20.201+1200 INFO eslegclient/connection.go:99 elasticsearch url: http://10.10.100.88:9200
2020-09-06T08:15:20.215+1200 INFO [publisher] pipeline/module.go:113 Beat name: AP08
2020-09-06T08:15:20.268+1200 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-09-06T08:15:20.269+1200 INFO instance/beat.go:450 filebeat start running.
2020-09-06T08:15:20.377+1200 INFO memlog/store.go:119 Loading data file of 'C:\ProgramData\filebeat\registry\filebeat' succeeded. Active transaction id=1933745420
2020-09-06T08:15:20.451+1200 INFO memlog/store.go:124 Finished loading transaction log file for 'C:\ProgramData\filebeat\registry\filebeat'. Active transaction id=1933749559
2020-09-06T08:15:20.500+1200 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 5381
2020-09-06T08:15:20.501+1200 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2020-09-06T08:15:20.513+1200 INFO beater/crawler.go:148 Stopping Crawler
2020-09-06T08:15:20.513+1200 INFO beater/crawler.go:158 Stopping 0 inputs
2020-09-06T08:15:20.513+1200 INFO beater/crawler.go:178 Crawler stopped
2020-09-06T08:15:20.513+1200 INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2020-09-06T08:15:20.513+1200 INFO [registrar] registrar/registrar.go:166 Ending Registrar
2020-09-06T08:15:20.623+1200 INFO [registrar] registrar/registrar.go:137 Registrar stopped
2020-09-06T08:15:20.655+1200 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":218,"time":{"ms":218}},"total":{"ticks":608,"time":{"ms":608},"value":608},"user":{"ticks":390,"time":{"ms":390}}},"handles":{"open":231},"info":{"ephemeral_id":"a3c45770-924a-4e38-a131-910ea16909bd","uptime":{"ms":1505}},"memstats":{"gc_next":35015200,"memory_alloc":18731448,"memory_total":88252664,"rss":72908800},"runtime":{"goroutines":31}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4}}}}}
2020-09-06T08:15:20.655+1200 INFO [monitoring] log/log.go:154 Uptime: 1.5072252s
2020-09-06T08:15:20.655+1200 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-09-06T08:15:20.656+1200 INFO instance/beat.go:456 filebeat stopped.
2020-09-06T08:15:20.658+1200 ERROR instance/beat.go:951 Exiting: Failed to start crawler: creating module reloader failed: fileset iis/var is configured but doesn't exist

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.