Is it a bad practice to stop ELK daily

nahiko2000 · March 6, 2020, 1:03pm

Hi!

I guess it is a bad practice to stop Elastic/Logstash/Kibana in the evening and start it in the morning, but it is not going to be used (we use it for logs, some logs would be ingested, but they can be ingested in the morning since Filebeat gets backpressure)

What are the reasons (please tell me as many as you can) why we should not stop it.

I know ".monitoring " indices get deleted at midnight and they would not get automatically deleted (can they be configured so they delete at some other time?)

Thanks a lot!

DavidTurner · March 6, 2020, 3:50pm

I don't know of any reasons why you should not stop your cluster overnight if you don't need it. It's not a very common practice, for sure, so maybe you will encounter some issues that nobody has seen before. If so, please report them.

When you say that some indices get deleted at midnight, what mechanism is deleting them? If you are using ILM to delete old indices then it should deal with the cluster being unavailable and catch up when you restart it in the morning. Other mechanisms might not be so robust.

nahiko2000 · March 7, 2020, 3:53pm

Hi!

The indices that are getting deleted at midnight are the ones Filebeat, Logstash, Kibana and Elasticsearch use for monitoring, indices whose names are ".monitoring-es-7-2020.03.06", ".monitoring-kibana-7-2020.03.06", ".reporting-2020.02.16", ".watcher-history-9-2020.03.06"...
Some of them, at least the ones which are ".monitoring..." get deleted at midnight, if Elasticsearch is not running at midnight, they will not get deleted, ever. Do you know of a way to configure Elastic so they get deleted at some other time?

Thanks a lot!!

DavidTurner · March 7, 2020, 5:40pm

Sure, but what is it that's supposed to be deleting these indices? Curator? ILM? Your own application?

nahiko2000 · March 10, 2020, 12:02pm

Hi!

These are ELK + Filebeat internal monitoring indices.
We do not use metricbeat, we use what Kibana calls "self monitoring".
These indices get deleted automatically when they get 7 days old. We can configure them to last more o less than 7 days changing the setting "xpack.monitoring.history.duration" but we do not know if we can set them to be deleted at some other schedule.

Thanks!

DavidTurner · March 10, 2020, 12:18pm

Thanks, that's helpful. TIL that xpack.monitoring.history.duration only cleans things up once per day, so this will indeed be problematic if your cluster isn't running at the time:

github.com

elastic/elasticsearch/blob/85b4bfe3ff119b3b7fa48631c6fc4a0bf9e1816d/x-pack/plugin/monitoring/src/main/java/org/elasticsearch/xpack/monitoring/cleaner/CleanerService.java#L241-L258


/**
 * Schedule task so that it will be executed everyday at the next 01:00 AM.
 */
static class DefaultExecutionScheduler implements ExecutionScheduler {


    @Override
    public TimeValue nextExecutionDelay(ZonedDateTime now) {
        // Runs at 01:00 AM today or the next day if it's too late
        ZonedDateTime next = now.toLocalDate()
            .atStartOfDay(now.getZone())
            .plusHours(1);
        // if it's not after now, then it needs to be the next day!
        if (next.isAfter(now) == false) {
            next = next.plusDays(1);
        }
        return TimeValue.timeValueMillis(Duration.between(now, next).toMillis());
    }
}

I think we can do better here, although maybe the answer is not to use this cleaner and instead to use ILM as I suggested above. If you can't use ILM, please open a Github issue about this -- maybe we can make the time of day configurable, or maybe there's some other solution.

nahiko2000 · March 10, 2020, 12:31pm

Thank you very much!! That helped a lot!!

Now that we see there is no other way, we will use ILM, Curator or a script to clean everything up.

Thanks!

nahiko2000 · April 1, 2020, 2:49pm

Hi again!!

We are having a problem with stopping ELK at night, we are getting a lot of duplicated documents, every time a log trace file rotates while ELK is stopped, when ELK starts again, every file from rotated file get ingested again.
I mean, we ingest for example "trace.log*", having 10 files for rotation, if "trace.log" rotates at night (while ELK is down) when it starts, trace.log, trace.log.1, trace.log.2, ..., trace.log.10 get ingested AGAIN, so we get thousands of duplicated documents.

Can you please look at this?
https://discuss.elastic.co/t/duplicated-documents-when-log-file-is-rotated-while-elk-is-down/225655

Thanks a lot!

system · April 29, 2020, 2:49pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.