There is some overlap between MLFlow and MLRun, but they have totally different goals. MLRun is an end to end orchestration layer for ML and MLOps. It’s not primarily a tracking system, though it does offer that functionality. MLFlow is a way to track your experiments, a component in the experimentation phase. There are also some ways to define metadata in MLFlow. However, there are many parts of the MLOps lifecycle that MLFlow doesn’t cover, like applying automation, automating serverless functions, running jobs, model monitoring, preparing data logic, and so on.
Airflow is a great tool for when you want to track experiments and show great charts showing experiment results but it’s not a tool that will save time getting to production.
MLRun is for what we call AutoMLOps, where the entire operationalization process is automated. MLRun uses serverless function technology: write the code once, using your preferred development environment and simple “local” semantics, and then run it as-is on different platforms and at scale. MLRun automates the build process, execution, data movement, scaling, versioning, parameterization, output tracking, CI/CD integration, deployment to production, monitoring, and more. MLRun provides an open pluggable architecture, so you have the option to use MLFlow (or any other tool) for the development side, and then use MLRun to automate the production distributed training environment without adding glue logic.