Unable to pick the data from NAS Drive


(Kumar) #1

We are unable to fetch data from NAS path.

We have given path as example \BNGWIDFL001\ECMLogs\Service.log

we can see filebeat is sending data and no errors found in filebeat logs, but as in elasticsearch we can see error logs generated.

[2017-04-03T16:56:28,526][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [bngwidap108.aonnet.aon.net] collector [indices-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:47) ~[elasticsearch-5.2.0.jar:5.2.0]

please let us know why logs not been indexed and cant able to view in kibana.


(Kumar) #2

please can anyone help me with above error.


(David Pilato) #3

May be providing the full elasticsearch logs could give a clue?


(Kumar) #4

Please find the full elasticsearch logs below.

[2017-04-04T13:28:06,255][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initializing ...
[2017-04-04T13:28:06,533][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] using [1] data paths, mounts [[New Volume (D:)]], net usable_space [147.4gb], net total_space [199.9gb], spins? [unknown], types [NTFS]
[2017-04-04T13:28:06,533][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-04T13:28:24,130][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] node name [bngwidap108.aonnet.aon.net], node ID [SgWDBkwLQ9KtIuf7vXQwbg]
[2017-04-04T13:28:24,145][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] version[5.2.0], pid[8104], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2017-04-04T13:28:28,294][WARN ][o.e.x.XPackPlugin        ] the [action.auto_create_index] setting is configured to be restrictive [.security,.monitoring*,.watches,.triggered_watches,.watcher-history*].  for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-YYYY.MM.dd] are allowed to be created
[2017-04-04T13:28:28,327][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [aggs-matrix-stats]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [ingest-common]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-expression]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-groovy]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-mustache]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-painless]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [percolator]
[2017-04-04T13:28:28,328][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [reindex]
[2017-04-04T13:28:28,329][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty3]
[2017-04-04T13:28:28,329][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty4]
[2017-04-04T13:28:28,329][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded plugin [x-pack]
[2017-04-04T13:28:44,423][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initialized
[2017-04-04T13:28:44,423][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] starting ...
[2017-04-04T13:28:44,860][INFO ][o.e.t.TransportService   ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9300}, bound_addresses {[::]:9300}
[2017-04-04T13:28:44,876][INFO ][o.e.b.BootstrapChecks    ] [bngwidap108.aonnet.aon.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-04T13:28:48,072][INFO ][o.e.c.s.ClusterService   ] [bngwidap108.aonnet.aon.net] new_master {bngwidap108.aonnet.aon.net}{SgWDBkwLQ9KtIuf7vXQwbg}{2AzEYyoCSoesKTYEHz4CsA}{10.209.68.107}{10.209.68.107:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-04T13:28:48,150][INFO ][o.e.h.HttpServer         ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9200}, bound_addresses {[::]:9200}
[2017-04-04T13:28:48,150][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] started
[2017-04-04T13:28:49,009][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:151) ~[elasticsearch-5.2.0.jar:5.2.0]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
	
[2017-04-04T13:28:54,555][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [bngwidap108.aonnet.aon.net] collector [index-recovery-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-04T13:28:54,570][ERROR][o.e.x.m.c.i.IndexStatsCollector] [bngwidap108.aonnet.aon.net] collector [index-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-04T13:28:54,617][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [bngwidap108.aonnet.aon.net] collector [indices-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
[2017-04-04T13:28:54,664][INFO ][o.e.x.m.e.Exporters      ] [bngwidap108.aonnet.aon.net] skipping exporter [default_local] as it is not ready yet
[2017-04-04T13:28:54,961][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:151) ~[elasticsearch-5.2.0.jar:5.2.0]

(David Pilato) #5

May be this is related?

[2017-04-04T13:28:28,294][WARN ][o.e.x.XPackPlugin        ] the [action.auto_create_index] setting is configured to be restrictive [.security,.monitoring*,.watches,.triggered_watches,.watcher-history*].  for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-YYYY.MM.dd] are allowed to be created

What are you elasticsearch.yml settings?


(Kumar) #6

Yes, i have used below lines in elasticsearch yml file.

action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*

May i know should i comment that line.?


(David Pilato) #7

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

May i know should i comment that line.?

Probably.


(Kumar) #8
i have commented the below line in elasticsearch yml file.
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*


i can see data is not visible in discover tab in kibana.

i can find there are errors in elasticsearch logs as below.

[2017-04-05T12:59:28,663][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initializing ...
[2017-04-05T12:59:28,772][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] using [1] data paths, mounts [[New Volume (D:)]], net usable_space [147.4gb], net total_space [199.9gb], spins? [unknown], types [NTFS]
[2017-04-05T12:59:28,772][INFO ][o.e.e.NodeEnvironment    ] [bngwidap108.aonnet.aon.net] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-05T12:59:42,231][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] node name [bngwidap108.aonnet.aon.net], node ID [SgWDBkwLQ9KtIuf7vXQwbg]
[2017-04-05T12:59:42,231][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] version[5.2.0], pid[7812], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_101/25.101-b13]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [aggs-matrix-stats]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [ingest-common]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-expression]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-groovy]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-mustache]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [lang-painless]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [percolator]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [reindex]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty3]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded module [transport-netty4]
[2017-04-05T12:59:44,573][INFO ][o.e.p.PluginsService     ] [bngwidap108.aonnet.aon.net] loaded plugin [x-pack]
[2017-04-05T12:59:53,769][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] initialized
[2017-04-05T12:59:53,769][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] starting ...
[2017-04-05T12:59:54,363][INFO ][o.e.t.TransportService   ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9300}, bound_addresses {[::]:9300}
[2017-04-05T12:59:54,363][INFO ][o.e.b.BootstrapChecks    ] [bngwidap108.aonnet.aon.net] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-04-05T12:59:57,510][INFO ][o.e.c.s.ClusterService   ] [bngwidap108.aonnet.aon.net] new_master {bngwidap108.aonnet.aon.net}{SgWDBkwLQ9KtIuf7vXQwbg}{FvSfLJj9QXqUF-DH6QNdSw}{10.209.68.107}{10.209.68.107:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-04-05T12:59:57,557][INFO ][o.e.h.HttpServer         ] [bngwidap108.aonnet.aon.net] publish_address {10.209.68.107:9200}, bound_addresses {[::]:9200}
[2017-04-05T12:59:57,557][INFO ][o.e.n.Node               ] [bngwidap108.aonnet.aon.net] started
[2017-04-05T12:59:57,870][WARN ][r.suppressed             ] path: /.reporting-*/esqueue/_search, params: {index=.reporting-*, type=esqueue, version=true}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:151) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:116) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-05T13:00:03,854][ERROR][o.e.x.m.c.i.IndexRecoveryCollector] [bngwidap108.aonnet.aon.net] collector [index-recovery-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.admin.indices.recovery.TransportRecoveryAction.checkGlobalBlock(TransportRecoveryAction.java:114) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-05T13:00:03,901][ERROR][o.e.x.m.c.i.IndexStatsCollector] [bngwidap108.aonnet.aon.net] collector [index-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
	at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.2.0.jar:5.2.0]
[2017-04-05T13:00:03,932][ERROR][o.e.x.m.c.i.IndicesStatsCollector] [bngwidap108.aonnet.aon.net] collector [indices-stats-collector] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:165) ~[elasticsearch-5.2.0.jar:5.2.0]
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:256) ~[elasticsearch-5.2.0.jar:5.2.0]

Continuously i can see failed to collect data in ES logs. Please help what needs to be done.

(David Pilato) #9

May be delete the current .reporting-* indices?

DELETE .reporting-*

(Kumar) #10
[quote="Sujith, post:8, topic:80992"]
reporting-*
[/quote]

Thanks dadoonet, i have removed .reporting-*

still i cant able to get logs from NAS path

I have given below lines in filebeat yml.
filebeat.prospectors:
- input_type: log
paths:
    - /var/log/*.log
    - \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log
    - D:\logs\*.log

document_type: log
scan_frequency: 5s

output.logstash:
  # The Logstash hosts
  hosts: ["10.209.68.107:5044"]
  index: "filebeat-%{+yyyy.MM.dd}"

Please let us know below NAS path is configured properly or not.
Your inputs helps us a lot.
- \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log

(David Pilato) #11

This is a filebeat question. Please ask in #beats:filebeat instead.


(system) closed #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.