Deploying Models
0 likes65 views
The document discusses machine learning inferencing, outlining the distinction between batch and real-time inference methods. Batch inference processes predictions in scheduled intervals, storing results for later use, while real-time inference offers immediate predictions for applications needing rapid responses. Best practices for implementing both methods in Azure Machine Learning, including compute options and deployment strategies, are provided.
Deploying Models
- 2. Eng Teong Cheah Microsoft MVP
- 3. Inferencing? In machine learning, inferencing refers to the use of a trained model to predict labels for new data on which the model has not been trained. Often, the model is deployed as part of a service that enables applications to request immediate, or real-time, predictions for individual, or small numbers of data observations. In Azure Machine Learning, you can create real-time inferencing solutions by deploying a model as a service, hosted in a containerized platform, such as Azure Kubernetes Services (AKS).
- 4. Deploying a Real-Time Inferencing Service
- 5. Machine learning inference during deployment When deploying your AI model during production, you need to consider how it will make predictions. The two main processes for AI models are: •Batch inference: An asynchronous process that bases its predictions on a batch of observations. The predictions are stored as files or in a database for end users or business applications. •Real-time (or interactive) inference: Frees the model to make predictions at any time and trigger an immediate response. This pattern can be used to analyze streaming and interactive application data.
- 6. Machine learning inference during deployment Consider the following questions to evaluate your model, compare the two processes, and select the one that suits your model: •How often should predictions be generated? •How soon are the results needed? •Should predictions be generated individually, in small batches, or in large batches? •Is latency to be expected from the model? •How much compute power is needed to execute the model? •Are there operational implications and costs to maintain the model?
- 7. Batch inference Batch inference, sometimes called offline inference, is a simpler inference process that helps models to run in timed intervals and business applications to store predictions. Consider the following best practices for batch inference: •Trigger batch scoring: Use Azure Machine Learning pipelines and the ParallelRunStep feature in Azure Machine Learning to set up a schedule or event- based automation. •Compute options for batch inference: Since batch inference processes don't run continuously, it's recommended to automatically start, stop, and scale reusable clusters that can handle a range of workloads. Different models require different environments, and your solution needs to be able to deploy a specific environment and remove it when inference is over for the compute to be available for the next model.
- 8. Real-time inference Real-time, or interactive, inference is architecture where model inference can be triggered at any time, and an immediate response is expected. This pattern can be used to analyze streaming data, interactive application data, and more. This mode allows you to take advantage of your machine learning model in real time and resolves the cold-start problem outlined above in batch inference. The following considerations and best practices are available if real-time inference is right for your model: •The challenges of real-time inference: Latency and performance requirements make real-time inference architecture more complex for your model. A system might need to respond in 100 milliseconds or less, during which it needs to retrieve the data, perform inference, validate and store the model results, run any required business logic, and return the results to the system or application.
- 9. Real-time inference •Compute options for real-time inference: The best way to implement real-time inference is to deploy the model in a container form to Docker or Azure Kubernetes Service (AKS) cluster and expose it as a web service with a REST API. This way, the model runs in its own isolated environment and can be managed like any other web service. Docker and AKS capabilities can then be used for management, monitoring, scaling, and more. The model can be deployed on-premises, in the cloud, or on the edge. The preceding compute decision outlines real-time inference.
- 10. Real-time inference •Multiregional deployment and high availability: Regional deployment and high availability architectures need to be considered in real-time inference scenarios, as latency and the model's performance will be critical to resolve. To reduce latency in multiregional deployments, it's recommended to locate the model as close as possible to the consumption point. The model and supporting infrastructure should follow the business' high availability and DR principles and strategy.
- 11. Create a real-time inference service https://ceteongvanness.wordpress.com/2022/11/01/ create-a-real-time-inference-service/