Huge size of log files

Elastic Stack Elasticsearch
1

Hi,

I have Elasticsearch installed as Windows Service. I noticed several times during last several days that Elasticsearch log files took whole free disk space. It seems to me it happens every several hours. I do not have big indexes at all, but have Kibana and Logstash on the same PC. I'm just creating simple index templates, delete them, create/delete index based on the template, index several lines from CSV file, run some queries with different analyzers, etc. File names are elasticsearch-service-x64-stdout.2016-09-19.log and elasticsearch.log located in elasticsearch\logs folder. The size of every file is about 16Gb right now. I just stop that Windows Service and remove log files. Otherwise they will take all free disk space. What can cause such grow of log files and how to prevent them from growing? Thanks.

2

What's in your log files?

3

Just found that the following exceptions repeats again and again:

[2016-09-20 22:30:46,567][WARN ][gateway ] [Silver Squire] [localhost:9200][0]: failed to list shard for shard_started on node [UQVwtGtJQP-CPJUF4w8fTQ]
FailedNodeException[Failed node [UQVwtGtJQP-CPJUF4w8fTQ]]; nested: RemoteTransportException[[Silver Squire][127.0.0.1:9300][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: InvalidPathException[Illegal char <:> at index 9: localhost:9200];
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:204)
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$800(TransportNodesAction.java:106)
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$2.handleException(TransportNodesAction.java:177)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:855)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:833)
at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:387)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: RemoteTransportException[[Silver Squire][127.0.0.1:9300][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: InvalidPathException[Illegal char <:> at index 9: localhost:9200];
Caused by: ElasticsearchException[failed to load started shards]; nested: InvalidPathException[Illegal char <:> at index 9: localhost:9200];
at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:154)
at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:59)
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:92)
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:228)
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:224)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 9: localhost:9200
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
at org.elasticsearch.env.NodeEnvironment$NodePath.resolve(NodeEnvironment.java:112)
at org.elasticsearch.env.NodeEnvironment$NodePath.resolve(NodeEnvironment.java:105)
at org.elasticsearch.env.NodeEnvironment.availableShardPaths(NodeEnvironment.java:678)
at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:125)
... 10 more

4

It looks like you have a directory or file named "localhost:9200". Is that the name you gave a node, cluster, or index?

5

Thanks Magnus for replying. It is quite possible that I created some object\resource with a wrong name which is not quote supported by the Elasticsearch. I reinstalled the latest version of the Elasticsearch search yesterday and there is no such issue. Thanks anyway for helping. I will reply if the issue comes back.