Unable to pick the data from NAS Drive

Sujith · April 3, 2017, 11:31am

We are unable to fetch data from NAS path.

We have given path as example \BNGWIDFL001\ECMLogs\Service.log

we can see filebeat is sending data and no errors found in filebeat logs, but as in elasticsearch we can see error logs generated.

[2017-04-03T16:56:28,526][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [bngwidap108.aonnet.aon.net] collector [indices-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:47) ~[elasticsearch-5.2.0.jar:5.2.0]

please let us know why logs not been indexed and cant able to view in kibana.

Sujith · April 4, 2017, 8:20am

please can anyone help me with above error.

dadoonet · April 4, 2017, 9:24am

May be providing the full elasticsearch logs could give a clue?

Sujith · April 4, 2017, 10:05am

Please find the full elasticsearch logs below.

[2017-04-04T13:28:06,255][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initializing ...
[2017-04-04T13:28:06,533][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] using [1] data paths, mounts [[New Volume (D:)]], net usable_space [147.4gb], net total_space [199.9gb], spins? [unknown], types [NTFS]
[2017-04-04T13:28:06,533][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-04T13:28:24,130][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] node name [bngwidap108.aonnet.aon.net], node ID [SgWDBkwLQ9KtIuf7vXQwbg]
[2017-04-04T13:28:24,145][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] version[5.2.0], pid[8104], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2017-04-04T13:28:28,294][WARN ][o.e.x.XPackPlugin        ] the [action.auto_create_index] setting is configured to be restrictive [.security,.monitoring*,.watches,.triggered_watches,.watcher-history*].  for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-YYYY.MM.dd] are allowed to be created
[2017-04-04T13:28:28,327][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [aggs-matrix-stats]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [ingest-common]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-expression]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-groovy]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-mustache]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-painless]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [percolator]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [reindex]
[2017-04-04T13:28:28,329][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty3]
[2017-04-04T13:28:28,329][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty4]
[2017-04-04T13:28:28,329][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded plugin [x-pack]
[2017-04-04T13:28:44,423][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initialized
[2017-04-04T13:28:44,423][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] starting ...
[2017-04-04T13:28:44,860][INFO ][o.e.t.TransportService   ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9300}, bound_addresses {[::]:9300}
[2017-04-04T13:28:44,876][INFO ][o.e.b.BootstrapChecks    ] [bngwidap108.aonnet.aon.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-04T13:28:48,072][INFO ][o.e.c.s.ClusterService   ] [bngwidap108.aonnet.aon.net] new_master {bngwidap108.aonnet.aon.net}{SgWDBkwLQ9KtIuf7vXQwbg}{2AzEYyoCSoesKTYEHz4CsA}{10.209.68.107}{10.209.68.107:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-04T13:28:48,150][INFO ][o.e.h.HttpServer         ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9200}, bound_addresses {[::]:9200}
[2017-04-04T13:28:48,150][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] started
[2017-04-04T13:28:49,009][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:151) ~[elasticsearch-5.2.0.jar:5.2.0]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
	
[2017-04-04T13:28:54,555][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [bngwidap108.aonnet.aon.net] collector [index-recovery-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-04T13:28:54,570][ERROR][o.e.x.m.c.i.IndexStatsCollector] [bngwidap108.aonnet.aon.net] collector [index-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-04T13:28:54,617][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [bngwidap108.aonnet.aon.net] collector [indices-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
[2017-04-04T13:28:54,664][INFO ][o.e.x.m.e.Exporters      ] [bngwidap108.aonnet.aon.net] skipping exporter [default_local] as it is not ready yet
[2017-04-04T13:28:54,961][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:151) ~[elasticsearch-5.2.0.jar:5.2.0]

dadoonet · April 4, 2017, 1:41pm

May be this is related?

[2017-04-04T13:28:28,294][WARN ][o.e.x.XPackPlugin        ] the [action.auto_create_index] setting is configured to be restrictive [.security,.monitoring*,.watches,.triggered_watches,.watcher-history*].  for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-YYYY.MM.dd] are allowed to be created

What are you elasticsearch.yml settings?

Sujith · April 4, 2017, 1:55pm

Yes, i have used below lines in elasticsearch yml file.

action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*

May i know should i comment that line.?

dadoonet · April 4, 2017, 2:08pm

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

May i know should i comment that line.?

Probably.

Sujith · April 5, 2017, 7:50am

i have commented the below line in elasticsearch yml file.
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*


i can see data is not visible in discover tab in kibana.

i can find there are errors in elasticsearch logs as below.

[2017-04-05T12:59:28,663][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initializing ...
[2017-04-05T12:59:28,772][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] using [1] data paths, mounts [[New Volume (D:)]], net usable_space [147.4gb], net total_space [199.9gb], spins? [unknown], types [NTFS]
[2017-04-05T12:59:28,772][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-05T12:59:42,231][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] node name [bngwidap108.aonnet.aon.net], node ID [SgWDBkwLQ9KtIuf7vXQwbg]
[2017-04-05T12:59:42,231][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] version[5.2.0], pid[7812], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [aggs-matrix-stats]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [ingest-common]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-expression]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-groovy]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-mustache]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-painless]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [percolator]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [reindex]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty3]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty4]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded plugin [x-pack]
[2017-04-05T12:59:53,769][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initialized
[2017-04-05T12:59:53,769][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] starting ...
[2017-04-05T12:59:54,363][INFO ][o.e.t.TransportService   ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9300}, bound_addresses {[::]:9300}
[2017-04-05T12:59:54,363][INFO ][o.e.b.BootstrapChecks    ] [bngwidap108.aonnet.aon.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-05T12:59:57,510][INFO ][o.e.c.s.ClusterService   ] [bngwidap108.aonnet.aon.net] new_master {bngwidap108.aonnet.aon.net}{SgWDBkwLQ9KtIuf7vXQwbg}{FvSfLJj9QXqUF-DH6QNdSw}{10.209.68.107}{10.209.68.107:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-05T12:59:57,557][INFO ][o.e.h.HttpServer         ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9200}, bound_addresses {[::]:9200}
[2017-04-05T12:59:57,557][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] started
[2017-04-05T12:59:57,870][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:151) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:116) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-05T13:00:03,854][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [bngwidap108.aonnet.aon.net] collector [index-recovery-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:114) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-05T13:00:03,901][ERROR][o.e.x.m.c.i.IndexStatsCollector] [bngwidap108.aonnet.aon.net] collector [index-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-05T13:00:03,932][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [bngwidap108.aonnet.aon.net] collector [indices-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:256) ~[elasticsearch-5.2.0.jar:5.2.0]

Continuously i can see failed to collect data in ES logs. Please help what needs to be done.

dadoonet · April 5, 2017, 9:46am

May be delete the current .reporting-* indices?

DELETE .reporting-*

Sujith · April 5, 2017, 10:14am

[quote="Sujith, post:8, topic:80992"]
reporting-*
[/quote]

Thanks dadoonet, i have removed .reporting-*

still i cant able to get logs from NAS path

I have given below lines in filebeat yml.
filebeat.prospectors:
- input_type: log
paths:
    - /var/log/*.log
    - \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log
    - D:\logs\*.log

document_type: log
scan_frequency: 5s

output.logstash:
  # The Logstash hosts
  hosts: ["10.209.68.107:5044"]
  index: "filebeat-%{+yyyy.MM.dd}"

Please let us know below NAS path is configured properly or not.
Your inputs helps us a lot.
- \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log

dadoonet · April 5, 2017, 11:49am

This is a filebeat question. Please ask in #beats:filebeat instead.

system · May 3, 2017, 11:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch errors Elasticsearch	16	16368	September 1, 2017
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized] Elasticsearch	4	1019	August 17, 2022
Elasticsearch Errors not indexing data Elasticsearch	3	1526	April 16, 2019
数据节点因滞后而掉出集群 Elasticsearch	3	393	July 24, 2023
Error in Winlogbeat logs Beats winlogbeat	11	2333	August 16, 2018

Unable to pick the data from NAS Drive

Related topics