Curator: How to run from AWS Lambda properly?

[rolodato (Rodrigo López Dato)]

Hi,

I'm fairly new to both Python and AWS Lambda, but I'm trying to get Curator running on it. This is the lambda function I'm using now, which works:

import os
import logging
import curator

logging.getLogger().handlers = []

def handler(event, context):
    os.chdir(os.environ["LAMBDA_TASK_ROOT"])
    try:
        curator.cli.main(["--config", "curator.yml", "actions.yml"])
    except Exception as e:
        logging.error(e)
        raise e
    finally:
        return {}

The only problem with this is that running the lambda multiple times will result in duplicate logs. For example:

1st invocation:

START RequestId: e8f05afa-64be-11e8-8c73-87c552395dec Version: $LATEST
2018-05-31 10:39:35,665 INFO      Preparing Action ID: 1, "create_index"
2018-05-31 10:39:35,865 INFO      Trying Action ID: 1, "create_index": Just testing
2018-05-31 10:39:35,865 INFO      Creating index "curator-10:39:35" with settings: {}
2018-05-31 10:39:36,261 INFO      Action ID: 1, "create_index" completed.
2018-05-31 10:39:36,261 INFO      Job completed.
END RequestId: e8f05afa-64be-11e8-8c73-87c552395dec
REPORT RequestId: e8f05afa-64be-11e8-8c73-87c552395dec	Duration: 672.10 ms	Billed Duration: 700 ms 	Memory Size: 128 MB	Max Memory Used: 30 MB	

2nd invocation:

START RequestId: f6ea15af-64be-11e8-99cb-099ed037a57c Version: $LATEST
2018-05-31 10:39:58,506 INFO      Preparing Action ID: 1, "create_index"
2018-05-31 10:39:58,506 INFO      Preparing Action ID: 1, "create_index"
2018-05-31 10:39:58,829 INFO      Trying Action ID: 1, "create_index": Just testing
2018-05-31 10:39:58,829 INFO      Trying Action ID: 1, "create_index": Just testing
2018-05-31 10:39:58,829 INFO      Creating index "curator-10:39:58" with settings: {}
2018-05-31 10:39:58,829 INFO      Creating index "curator-10:39:58" with settings: {}
2018-05-31 10:39:59,193 INFO      Action ID: 1, "create_index" completed.
2018-05-31 10:39:59,193 INFO      Action ID: 1, "create_index" completed.
2018-05-31 10:39:59,193 INFO      Job completed.
2018-05-31 10:39:59,193 INFO      Job completed.
END RequestId: f6ea15af-64be-11e8-99cb-099ed037a57c
REPORT RequestId: f6ea15af-64be-11e8-99cb-099ed037a57c	Duration: 790.61 ms	Billed Duration: 800 ms 	Memory Size: 128 MB	Max Memory Used: 31 MB	

Further invocations just add more duplicate logs, until the Lambda is run in a new instance. I've tried different variations on setting handlers = [] or propagate = False on loggers for the CLI to no avail.

Is this (calling curator.cli.main() with arguments) the correct way to run Curator on AWS Lambda? If so, are there any workarounds for the duplicate logs?

Thanks!

[rolodato (Rodrigo López Dato)]

Update: I've managed to solve the duplicate logging with this Lambda function:

import os
import logging
import curator

def handler(event, context):
    logging.getLogger().handlers = []
    logging.getLogger(curator.__name__).handlers = []
    try:
        curator.cli.main(["--config", "curator.yml", "actions.yml"])
    except Exception as e:
        logging.error(e)
        raise e
    finally:
        return {}

I'm still curious if this is the best way to use Curator from Lambda. Thanks!