Easy to hit Seach Queue number in setting for watching a time series log

Elastic Stack Elasticsearch
1

I used ES to store structured time based log. So the index will be like XX-2015-06-07 , XX-2015-0608, ...
When using ES watcher for notification, I'd like watcher to check the status every 5 minute.

Is there a way that I can limit the search indices to be only the latest two days?

Because when I use all "XX-*" as search index, it will be very easy to hit the search queue limit(mine here is by default 1000, and sharding is 5)

2

Not at the moment, though we are working on a feature that will enable you to do just that (basically define the index name pattern for the indices you wish to search). For now you have two options:

  1. search all indices
  2. create a rolling alias that continuously points to the last 2 indices and search on that.
3

Thanks for your quick reply, Uri~ It is good to know that you are working on that.

Now I actually use option 1, to search all indices. And it will occur search failures on some shards.
About option 2, creating rolling alias to the latest 2 indices.
er... I'm a newbie for ELK and watcher. So can I ask here how can I do it? The rolling alias can be pointed to latest 2 indices automatically?(I didn't find the way on www.elastic.co ... )

4

sure... first, you can read here and here about aliases.

In short, you have an API that enables you to create/update aliases to indices. At the moment, there's nothing in elasticsearch that will automatically do that for you, so you need to make sure to call these APIs at the right times. For example, you can update an alias that points to the last two indices (assuming daily indices), by removing the index that is associated with two days ago, and adding a new index that is associated with today:

POST _aliases
{
    "actions" : [
        { "remove" : { "index" : "logstash-2015.06.06", "alias" : "last_two_days" } },
        { "add" : { "index" : "logstash-2015.06.08", "alias" : "last_two_days" } },
    ]
}

Calling this every midnight (again, assuming daily indices), the last_two_days alias will always point to the two indices representing the last two days. Now... here's the cool part, normally I'd tell you use curator or to run a cron job that does this automatically. But why use cron if you already have watcher... here's a watch that will take care of it for you:

PUT _watcher/watch/roll-logstash-indices-aliases
{
  "metadata" : {
    "index_pattern" : "'logstash-'YYYY.MM.dd",
    "rollover" : "1d"
  },
  "trigger" : {
    "schedule" : { "daily" : { "at" : "midnight" }}
  },
  "transform" : {
    "script" : "def rolloverMillis = org.elasticsearch.common.unit.TimeValue.parseTimeValue(ctx.metadata.rollover, null).millis(); def pattern = org.elasticsearch.common.joda.time.format.DateTimeFormat.forPattern(ctx.metadata.index_pattern); return [ 'rollover_millis' : rolloverMillis, 'remove-index' : pattern.print(ctx.trigger.scheduled_time.minus(rolloverMillis * 2 + 1000)), 'add-index' : pattern.print(ctx.trigger.scheduled_time)];"
  },
  "actions" : {
    "update-alias" : {
      "webhook" : {
        "method" : "POST",
        "host" : "localhost",
        "port" : 9200,
        "path" : "/_aliases",
        "body" : {
          "inline" : {
            "actions" : [
              { "remove" : { "index" : "{{ctx.payload.remove-index}}", "alias" : "last_two_indices" } },
              { "add" : { "index" : "{{ctx.payload.add-index}}", "alias" : "last_two_indices" } }
            ]
          }
        }
      }
    }
  }
}

In the above watch, I defined the rolling interval and the logstash index name pattern in the metadata (so if you plan to change the rolling nature of the indices, these along with the schedule are the only parameters you'll need to change)... currently the rollover is set to 1 day and the index name pattern is set appropriately (so it'll change daily to indicate the associated date).

The transform takes care of computing the names of the indices that need to be deleted and added.

The webhook action calls the elasticsearch aliases API to add/remove the appropriate indices.

The schedule is set to run every midnight (appropriate for daily rollover).

hope it helps

5

Thank you very much for your detailed answer~ Uri~
I have done to create last_2_days alias. It really helped me.

6

Hi all,

I have to do indexing everyday at night. Without breaking the current application I have to use aliases by scheduling a job. Can someone let me know how to create a job for creating an alias and adding the new alias and removing the old one if the indexing is successful. Else keeping the old index as it is.

Eg : If my current alias is alias1 then new alias should be aliase2 and adding this and removing the aliase1 viceversa.

Thanks,
Vani Aravinda