Advanced Machine Learning with Databricks Training in Ireland

This course is aimed at data scientists and machine learning practitioners and consists of two, four-hours modules.

Participants should have:

• Basic knowledge of data science and machine learning concepts (e.g., classification and regression models).
• Familiarity with common ML evaluation metrics (e.g., F1-score).
• Experience with Python and ML libraries (e.g., scikit-learn, XGBoost).
• Intermediate-level knowledge of ML development and the use of Git for ML projects.

If you do not have one or more of the pre-requisites QA recommends:

• QADSMLP - Data Science and Machine Learning with Python
• DBASP - Apache Spark Programming with Databricks

Target Audience

This course is designed for:

• Data scientists and ML engineers who want to scale machine learning workflows with Databricks.
• MLOps practitioners aiming to streamline ML lifecycle management, testing, and deployment.
• AI/ML professionals implementing CI/CD, model monitoring, and production-ready ML systems.

Machine Learning at Scale

In this course, you will gain theoretical and practical knowledge of Apache Spark’s architecture and its application to machine learning workloads within Databricks. You will learn when to use Spark for data preparation, model training, and deployment, while also gaining hands-on experience with Spark ML and pandas APIs on Spark. This course will introduce you to advanced concepts like hyperparameter tuning and scaling Optuna with Spark. This course will use features and concepts introduced in the associate course such as MLflow and Unity Catalog for comprehensive model packaging and governance.

Advanced Machine Learning Operations

In this course, you will be provided with a comprehensive understanding of the machine learning lifecycle and MLOps, emphasizing best practices for data and model management, testing, and scalable architectures. It covers key MLOps components, including CI/CD, pipeline management, and environment separation, while showcasing Databricks’ tools for automation and infrastructure management, such as Databricks Asset Bundles (DABs), Workflows, and Mosaic AI Model Serving. You will learn about monitoring, custom metrics, drift detection, model rollout strategies, A/B testing, and the principles of reliable MLOps systems, providing a holistic view of implementing and managing ML projects in Databricks.

Machine Learning at Scale

Machine Learning Development with Spark

• A Brief Overview of Spark Architecture for Machine Learning
• Introduction to Spark ML for Model Development
• Model Tracking and Packaging with MLflow and Unity Catalog on Databricks
• Model Development with Spark

Model Tuning with Optuna on Spark

• Overview of Hyperparameter Tuning
• Introduction to Optuna on Spark
• Model Tuning with Optuna

Advanced Machine Learning Operations

Overview of Machine Learning Operations on Databricks

• Review of MLOps
• Streamlining Development to Deployment

Continuous Workflows for Machine Learning Operations

• Streamlining MLOps
• Streamlining MLOps with Databricks

Testing Strategies with Databricks

• Automate Comprehensive Testing
• Model Rollout Strategies with Databricks

Model Quality and Lakehouse Monitoring

• Introduction to Monitoring
• Lakehouse Monitoring

Streamlining Multiple Environment Deployments - DABsBuild ML assets as CodeCourse Summary and Next Steps