This repository includes a AWS Cloud Development Kit template for quickly building the pipeline

This solution introduces an automated solution for monitoring and evaluating the performance of time-series models, and creating new model using Amazon SageMaker Autopilot, specifically tailored for applications like day-ahead power forecasting for solar panels. Given the critical reliance on accurate forecasts in such applications, maintaining model performance is important.Currently, there are significant challenges as the existing Amazon SageMaker Model Monitor do not support time-series data monitoring. This solution intends to address this gap by automating the continuous evaluation of model outputs against daily ground truth data, identifying performance degradation swiftly.

When a model's performance drops below predefined thresholds, this pipeline automatically initiates Autopilot training processes to ensure predictions remain reliable and robust with new model. This not only enhances the trust in model outputs but also significantly reduces the manual effort required from data scientists and business owners. Following retraining, the system sends notifications to data scientists, allowing them to review and decide whether to deploy the updated model or continue with the existing one.

• Include inference job in the pipeline
• Support more objective metrics

New version will be released in July, click here fore more information

• Simplified the new model evaluation by directly using the object metric value from Autopilot job
• Only use RMSE as example metric for evaluation and training. If you want to use other metric, replace this code block in evaluation function

1. Architecture
2. Resources
3. Deployment
4. New Version

1. Generate synthetic dataset in jupyter notebook and upload data to S3 bucket
2. S3 object create event triggers Lambda function
3. The Lambda function processes the event and start state machine
4. Download ground truth from s3://<your-bucket>/data/hist and predicted result from s3://<your-bucket>/data/pred, calculate the RMSE of a certain day's prediction, and share the result
5. If perform well comparing with threshold, keep the current model and end the workflow, otherwise start to train new model
6. Start new Autopilot job vis calling create_auto_ml_job_v2
7. Share the Autopilot job result and current model performance to data scientist for further investigation

This is a list of the majority of the AWS services used in this project:

Replace [email protected] with your email address to receive the notifcation of model performance

$ cdk deploy -c [email protected]

Once the deployment of SNS completed, you should receive a email notification. Please click Confirm subscription for further notification.

Execute the solar-power-synthetic-data-generation notebook

You can find a running state machine in AWS Step Function Console once all cells ran:

Another email will be sent to your email address once the state machine completed. You can decide wheather to promote the new model or keep the current one after reviwing.

First, empty the S3 bucket.

To dispose of the stack afterwards, run the command:

$ cdk destroy

1. Generate synthetic dataset in jupyter notebook and upload data to S3 bucket. On Upload a wrokflow is triggered to generate parameters for evaluation, pulling these from Parameter Store.
2. Using a SageMaker Batch Transform Job, Predictions are generated for evaluation from the ground truth dataset. A Time Series evaluation lambda calculates the RMSE for our Model.
3. If our Model metric meets our business metric threshold, we update our registered model's metric metadata in the SageMaker Model Registry 4.If not, we generate a Model using the SageMaker AutoML V2 API. We then evaluate this model using a Batch Transform Job.
4. If the new model, performs better than the current model, a Model Package is registered for review in the Model Registry
5. Once our, evaluation is complete, we notify the relevant stakeholder to review the changes in model performance for potential approval.

• cdk ls list all stacks in the app
• cdk synth emits the synthesized CloudFormation template
• cdk deploy deploy this stack to your default AWS account/region
• cdk diff compare deployed stack with current state
• cdk docs open CDK documentation

Enjoy!

See CONTRIBUTING for more information.

This library is licensed under the MIT-0 License. See the LICENSE file.