Je n'ai plus accès à mes anciens index ES

Elastic Stack Kibana
1

Bonsoir,

j'ai installé Syslog-ng/Elasticsearch/Kibana et je viens de me rendre compte que lorsque je filtre les logs via la date dans kibana il ne m'affiche plus les logs d'hier ou d'il y a 2/3 jours, il ne m'affiche que ceux d'aujourd'hui.

J'ai remarqué que j'avais ce message affiché dans les logs et que la cause du problème vient du champs ISODATE qui n'existe pas mais je ne sais pas comment faire pour régler le problème:

[2017-02-09 17:37:13,760][DEBUG][action.fieldstats ] [thales5] [settings][3], node[dASTOYSMSUeoK8Wpo502OA], [P], v[164], s[STARTED], a[id=wIz0Yp1RTU678xL7RJnsxw]: failed to execute [org.elasticsearch.action.fieldstats.FieldStatsRequest@d84f73a]
RemoteTransportException[[thales8][127.0.0.1:9302][indices:data/read/field_stats[s]]]; nested: IllegalArgumentException[field [ISODATE] doesn't exist];
Caused by: java.lang.IllegalArgumentException: field [ISODATE] doesn't exist
at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:166)
at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:54)
at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:282)
at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:278)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Merci

2

Pourtant lorsque je regarde le mapping de mon index il y a bien le champs ISODATE de définie:

3

Sorry, unfortunately this forum is pretty much English-only, so it might be hard for you to get an answer with your French post.

I pasted your comment into Google Translate and I think I got the gist that you are trying to see logs for a certain date using a filter. This is what I got:

I installed Syslog-ng / Elasticsearch / Kibana and I just realized that when I filter the logs via the date in kibana it no longer shows me the logs of yesterday or 2/3 days ago , It only shows me those of today.

I noticed that I had this message displayed in the logs and that the cause of the problem comes from the ISODATE field that does not exist but I do not know how to solve the problem:

I would guess you have ISODATE selected as the time field in your Kibana Index Pattern, and you are using the timepicker in the top-right of the browser to filter the date. But your ISODATE fields might be just all null-values. Try doing a search in Console to find out if those fields are what you expect them to be.

4

Hello Tim,

First, thank you for your answer
Secondly, sorry for my english

And Yes i have ISODATE selected as time field but for the moment i use 2 index pattern:

  • one where i define normal index pattern that respect to the ISODATE and this his index pattern shows me the logs of the day

-and the other it is a index pattern which regroup all the logs but that is not in relation to time. I deselectd the case in red.

image.png1234×614 50.5 KB

My field ISODATE is not null :

How can I change the field time field and put something other than ISODATE?

Thank you

5

It's not a problem at all!

It looks like you have a strange index pattern that's based on * which I am guessing is the cause of the problems. If you have that index pattern selected and try to do a time-based filter, it will match against indices which are not time-based and don't have an ISODATE field, such as the system index for Kibana called .kibana.

Maybe your time-based index pattern could be created using the pattern syslog-ng_* and your non-timebased index pattern for regrouping the same data could be created using the pattern syslog-ng*. You would just have to remember when choosing the Index Pattern in Discover and Visualize that the _ at the end of the pattern name represents one set up for time-based usage.

6

yes i have already done what you propose, put "syslog-ng_* " for the index using time-based And "syslog-ng* " for all index without the time based.

But when i had installed Syslog Elasticsearch and Kibana for the first time and that i defined index pattern in kibana, i put "syslog-ng_* " using time-based and when i went in the discover tab i had all my index, all data when I filter by date I had all indexes from the beginning and that with one index pattern.

Now i must create 2 index pattern one for all data and another for index using time-based.

Maybe i have too much data so kibana can show me that the data of the day.

Thank you

7

I hope that removing the * index pattern at least took care of the errors you saw in the Elasticsearch logs that you mentioned in your first post.

Now as I understand it, the problem comes down to looking at a time-based index in Kibana with a lot of data, applying a filter on date against it, but not seeing the filter applied correctly - you still see documents that don't match that filter? If I have it correct, then can you walk me through how you are applying this filter? Is it using the query bar above the table in Discover or the timepicker control in the upper right corner of the browser?

There shouldn't be a problem with the amount of data for Kibana to handle. Everything goes through Elasticsearch so if you had a capacity problem you would see the Elasticsearch report problems with failed shards or the process run out of memory with the given JVM heap size.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.