Amazon SageMaker Autopilot experiments run with Ensemble training mode provide additional metrics and visibility into the AutoML workflow
Amazon SageMaker Autopilot now provides insights into the underlying workflow for each trial within a SageMaker Autopilot experiment launched with ensemble training mode. SageMaker Autopilot ranks a list of machine learning (ML) models by inference latency i.e. the time one has to wait to get prediction result from a real time endpoint to which the model is deployed, and objective metrics such as accuracy, precision, recall, and area under the curve (AUC) in the model leaderboard. SageMaker Autopilot automatically builds, trains and tunes the best ML models based on your data, while allowing you to maintain full control and visibility.
Amazon SageMaker Autopilot now provides insights into the underlying workflow for each trial within a SageMaker Autopilot experiment launched with ensemble training mode. SageMaker Autopilot ranks a list of machine learning (ML) models by inference latency i.e. the time one has to wait to get prediction result from a real time endpoint to which the model is deployed, and objective metrics such as accuracy, precision, recall, and area under the curve (AUC) in the model leaderboard. SageMaker Autopilot automatically builds, trains and tunes the best ML models based on your data, while allowing you to maintain full control and visibility.
Amazon SageMaker Autopilot recently added a new ensemble training mode powered by AutoGluon. In the ensemble training mode, multiple trials with different combinations of a subset of algorithms and AutoGluon configuration parameters are executed. Until now, only a single model from each trial run was returned as a trial output and was ranked by the objective metric on the model leaderboard. Starting today, SageMaker Autopilot experiments with ensemble training mode will not only provide increased visibility into the autoML experiment by listing all underlying set of base learner models that were run within each trial, but will also use both the best objective metrics and lowest inference latency to select the best model candidate for an experiment. As an example, if two model candidates for a binary classification problem type have a similar f1 score objective metric of 0.678 but an inference latency of 0.43 secs and 0.39 secs respectively, SageMaker Autopilot will rank the latter as the best model in the leaderboard.
To get started, update Amazon SageMaker Studio to the latest release and launch SageMaker Autopilot either from SageMaker Studio Launcher or APIs. To learn more on how to update SageMaker Studio, please see documentation.
The inference latency metric feature and visiblity into base learner models is now available in all regions where SageMaker Autopilot is available. To get started, see Creating an Experiment with Autopilot and SageMaker Autopilot API reference. To learn more, visit the SageMaker Autopilot product page.