Error sending data to elasticsearch

Elastic Stack Elasticsearch
1

I have come accorss with the following error. Is there anyone who have ideas with that?

2016/06/03 10:49:02.295137 client.go:190: INFO Bulk item insert failed (i=49, status=503): {"type":"unavailable_shards_exception","reason":"[topbeat-2016.06.03][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [topbeat-2016.06.03] containing [10] requests]"}

2

A 503 can indicate an overload which means the thread pools are busy and so the request is rejected.

3

Yes I found it is the disk size problem
What files, like old data, and where should I delete in elasticsearch to make the beats feed data again?
And how can I make the data to just keep for some time and delete automatucally after a certain period as they become old data

4

Hi,

You can use curator https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html to schedule delete

There are some example https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

You might want to delete index older than 'x' days for instance

5

When I tried to run the command, it comes across with the following issues.

[Administrator@localhost elasticsearch]$ curator --host 172.16.1.245 delete indices --older-than 2 --time-unit days --timestring '%Y.%m.%d'
Traceback (most recent call last):
File "/usr/bin/curator", line 5, in
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

6

I am now coming across with the problem like that

Exception in thread "main" java.nio.file.FileSystemException: /tmp/2427469249444301473.tmp: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
at java.nio.file.Files.createTempFile(Files.java:897)
at org.elasticsearch.bootstrap.Security.selfTest(Security.java:352)
at org.elasticsearch.bootstrap.Security.configure(Security.java:124)
at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:196)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.