Kibana 3 – Not Displaying Historical Logs

Elastic Stack Kibana
1

Hello, I have an ELK server with the below stack versions:

K: 3.1.2

E: 1.4.3

L: 1.4.2

I have an issue where I cannot view historical logs in
Kibana, this seems to be random as I was able to view the data on that date in
the past.

I have indices at /var/lib/elasticsearch/elasticsearch/nodes/0/indices
right back to 01/09/2015 (logstash-2015.01.09) and all the way down to todays
date, where I can only see the last weeks’ worth of logs.

Where am I best starting looking to resolve this problem? I
can see all my index names via curl http://localhost:9200/_aliases?pretty=1

Many thanks

EDIT:

I had two nodes that did this and I was able to
resolve it on one. It was receiving the below in the
/var/log/elasticsearch/elasticsearch.log:

[Black Tarantula] Caught exception while handling client
http traffic, closing connection [id: 0xdc2765c4, /127.0.0.1:57760 =>
/127.0.0.1:9200]

org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException:
An HTTP line is larger than 4096 bytes.

I was able to edit the required line in the
elasticsearch.yml but it didn’t resolve it for the other node, this one no
longer receives the above URL issue but receives the below now:

[DEBUG][action.search.type ] [Feron] All shards failed for phase:
[query]

2

Can anyone recomend a solution?

3

There should be something in the logs to help you figure this out.

Also, you should really upgrade, you've lagging behind current releases :slight_smile:

4

Yeah I have a build that runs the latest versions of all
stack apps, just this one is production right now though. I’m looking through
google trying to find some knowledge so just posting here too to see if an
answer can be given quicker.

5

There should be more in your logs.

Maybe if you can gist/pastebin/etc them we can have a look?

6

My issues relates to a large number of shards not being
allocated, for example:

logstash-2015.06.13 4 p UNASSIGNED
logstash-2015.06.13 4 r UNASSIGNED
logstash-2015.06.14 2 p UNASSIGNED
logstash-2015.06.14 2 r UNASSIGNED
logstash-2015.06.14 0 p UNASSIGNED
logstash-2015.06.14 0 r UNASSIGNED
logstash-2015.06.14 3 p UNASSIGNED
logstash-2015.06.14 3 r UNASSIGNED

not sure exactly why but I’ve been trying to re-allocate
them but I’m running into this issue below:

[NO(shard cannot be allocated on same node
[gMnx09_-TMqDfLJe5NkJWQ] it already exists on).

So I can’t re-assign onto the same node obviously but how do
I change the node?

7

You mentioned you had two nodes, do you only have one now?

8

I have only one cluster with the issue, the other one was fixed as the issue was something
else.