Logstash success output notification

[shane_lee]

Hi all,

I was wondering is it possible to send a message to SNS topic after successful bulk index to elasticsearch. Same for failure also.

I have attached a high level solution design for reference.

Thanks,
Shane.

[warkolm (Mark Walkom)]

If it's a once off thing you could build a script around a single call to LS that does it.
But it's not possible natively in LS.

[shane_lee]

Hi Mark,

Its not a once off thing. It will be a scheduled bulk index using logstash.

We are not reindexing, so we want to try and streamline the process by automatically switching alias after successful index using curator for example.

Is this possible?

Are output plugins sequential or async? Could i for example send a message to SNS after elasticsearch output?

[warkolm (Mark Walkom)]

Outputs are not sequential, they are parallel.

[shane_lee]

Ok. So what you recommend? Create a custom script to notify after bulk index?

[warkolm (Mark Walkom)]

Yes, but you can't keep LS running and have that, you'd nee to run the script, do the bulk, end LS and then rerun the script.

[shane_lee]

Is it possible to raise this as a feature request. Or does xpack provide notifications?

[warkolm (Mark Walkom)]

You can make a FR, sure

There is Alerting functionality, however I don't know if you would want to do per bulk notifications as that is likely to become overloading. Perhaps there is a better way to provide whatever you want to do?

[shane_lee]

Scenario is:

I bulk index (for example once a week) using logstash. It creates a new index every time.

Upon success, I would like to trigger a notification with the new index name.

I then have a curator (as lambda fn potentially) that receives this pushed message and switches the alias over from the old index to the new and delete the old index.

This a common flow!?

[theuntergeek (Aaron Mildenstein)]

@shane_lee

I then have a curator (as lambda fn potentially) that receives this pushed message and switches the alias over from the old index to the new and delete the old index.

See the Curator Rollover Action.

[shane_lee]

Hi @theuntergeek, long time no hear!

I had a look at rollover before, but I think thats related more to switching alias based on age or number of documents. Correct me if Im wrong.

Our data is a full refresh that we consume from an RDS. When logstash creates the new index and bulk indexes the data, I want some way of switching over the alias and deleting the old index.

We are using the elasticsearch managed service from AWS and are signing our requests.

I have attached the sample output plugin for your convenience:

amazon_es {
hosts => ["xxxxxxxx.ap-southeast-2.es.amazonaws.com"]
region => "ap-southeast-2"
index => "postaladdress-2017-05-15"
flush_size => "30000"
document_type => "postaladdress"
document_id => "%{delivy_point_id}"
template_name => postaladdress
template_overwrite => true
manage_template => true
template => "/usr/local/docker/logstash/migration/address-mapping-template.json"
}

What do you think?

[theuntergeek (Aaron Mildenstein)]

@shane_lee

I want some way of switching over the alias and deleting the old index.

This is exactly what the Rollover API does (minus the deleting part). But where the plain Rollover API ends is where Curator begins. A Curator action file could do the rollover as action 1, and then action 2 could be deleting indices (matching the pattern of the indices behind the alias) older than 1 day, or something like that. Since the new index would only be a matter of moments old, the recently rolled over index would be found by age, and deleted. An example configuration might look like this:

---
actions:
  1: 
    action: rollover
    description: Rollover the index behind "my_alias" when it is older than 7 days
    options:
      name: my_alias
      conditions:
        max_age: 7d
  2:
    action: delete_indices
    description: Delete indices matching pattern index-0000* older than 5 days
    options:
      ignore_empty_list: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: index-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
    - filtertype: age
      source: creation_date
      direction: younger
      unit: days
      unit_count: 7
      exclude: True

Update: I updated the filters to prevent younger indices from being deleted accidentally, and added the direction flag, which was unintentionally omitted.

[shane_lee]

Hi @theuntergeek,

Thanks for getting back to me.

The config above for rollover specifies the name of the alias to rollover. How does it know what index to switch to?

For example if my pattern for index name is index => "postaladdress-%{+yyyy.MM.dd}" in logstash.

Index 1 is postaladdress-2017.05.18 . It has an alias of postaladdress

Index 2 created is postaladdress-2017.05.25.

Will action 1 rollover alias from index postaladdress-2017.05.18 to postaladdress-2017.05.25?

From reading the doc, rollover creates a new index based on pattern. That is what I am uncertain of.

What I have done in the past is atomic operation of switching alias using alias API.
I know that curator handles alias API too.

Example:

POST /_aliases
{
"actions": [
{ "remove": { "index": "location1", "alias": "location" }},
{ "add": { "index": "location2", "alias": "location" }}
]
}

Let me know your thoughts.

Thanks,
Shane.

[shane_lee]

Ive tested it out:

> POST /postaladdress/_rollover/postaladdress-new-index
> {
>   "conditions": {
>     "max_docs":  1
>   }
> }

Get 400 bad request as the index has already been created.

I will have created my new index and indexed the documents using logstash.

I believe what I need is alias API and not rollover.

[theuntergeek (Aaron Mildenstein)]

To use Rollover, you have to change your index pattern. The official documentation provides this example:

PUT /logs-000001 
{
  "aliases": {
    "logs_write": {}
  }
}

# Add > 1000 documents to logs-000001

POST /logs_write/_rollover 
{
  "conditions": {
    "max_age":   "7d",
    "max_docs":  1000
  }
}

It also provides for naming the new index (there's a ticket for this for Curator):

POST /my_alias/_rollover/my_new_index_name
{
  "conditions": {
    "max_age":   "7d",
    "max_docs":  1000
  }
}

Of course, if you've renamed the first index, you could just keep the new indices incrementing, so you'd only have to do that once, unless you wanted a named pattern.

Summary: What I'm really suggesting is that you stop naming your indices with the date and start with -00001 and let them increment automatically via the Rollover API. Then you wouldn't need to re-alias every single day.

[shane_lee]

@theuntergeek

It also provides for naming the new index

I think we are not on the same page!!

I actually dont want to 'rollover' and create a new index.

Logstash (on a schedule) will create a new index using my predefined index mapping template. And then bulk index.

@shane_lee

template_name => postaladdress
template_overwrite => true
manage_template => true
template => "/usr/local/docker/logstash/migration/address-mapping-template.json"

All I want is some way to switch the alias over and delete the old index after logstash completes.

Rollover creates the new index based on certain conditions and switches the alias. I dont want that.

I am thinking i can use node or python to execute a shell command and call logstash. on completion i can then use curator to switch the alias and delete the old index.

Thats why Im thinking of using two actions in curator:

1. alias
2. delete_indices

Thanks,
Shane.