Version 3.4.0 Release Notes

Release Date: 2 May 2022

On This Page

This document outlines the main additions and changes to the Iguazio MLOps Platform ("the platform") in version 3.4.0.

Tech Preview Note
The "[Tech Preview]" notation marks features that are included in the release but have not undergone comprehensive quality assurance testing, and therefore should not be used in production. Contact support@iguazio.com for more details.

Beta Note
The "[Beta]" notation marks features that are not officially supported in this release.

Highlights

  • Trino (formerly Presto) is upgraded to 370.
  • New report that displays the consumer groups / v3io streams. See Consumer groups report.
  • Real-time pipelines are displayed in the UI. See Real-time pipelines. [Tech Preview]
  • Configuration of pod priority for the MLRun services and Nuclio functions in the UI. See Pod priority.
  • Configuration of spot nodes for all functions, when creating a job, rerunning an existing job, and creating an ML functions. See Spot nodes.
  • Configuration of CPU, GPU, and memory default limits for user jobs. See User job default limits.
  • Improve service deployment upon changes. See Modifying a service.

Managed Application Services

Jupyter Notebook

  • You can install packages on Jupyter, by using the iguazio user to perform apt-get using sudo. Installed packages are not persistent and are stored on the local container fs. Upon Jupyter restart/scale-down etc. those changes are lost. To make the packages persistent, add the package installation commands to your init script.

MLRun

  • The Feature store supports Spark engine ingestion from Snowflake.
  • Enhanced masking of sensitive data in the UI.
  • Setting owner and members have moved to a dedicated project Settings section in the UI.
  • The dataset tab is now in the Projects main menu (was previously under the Feature store).

Nuclio Serverless Functions

  • Nuclio integrates with Confluent (a managed Kafka service). [Tech Preview]
  • Within a Nuclio function, multiple volumes can use the same name.
  • Python 3.6 will soon be deprecated as a runtime for Nuclio.

Presto and Hive

  • Trino (formerly Presto) is upgraded to 370.

Security and User Management

  • The IT Admin management policy has rights to change the status of a cluster from offline to online.
  • The Project Admin management policy was renamed to Project Security Admin.

Backup, Recovery, and High Availability (HA)

  • If a cluster goes offline, the system automatically tries to move it to online. (If the cluster was in a degraded state when it went offline, it remains in a degraded state when it comes online.)

Dashboard (UI)

Projects

  • The Project Monitoring report has a new tile named Consumer groups (v3io streams) that shows the total number of consumer groups, with drill-down capabilities for more details.
  • There is a new tab under Projects | Models named Real-time pipelines that displays the real time pipeline graph, with a drill-down to view the steps and their details. [Tech Preview]

Miscellaneous

  • The Iguazio version in the About window now includes the patch version. For example, 3.4.0, instead of the previous 3.4.
  • The Support logs tab is now in the Clusters (was previously in the Data cluster tab).

Services

  • You can configure Pod priority for the MLRun services and Nuclio functions in the UI (previously it was configurable by SDK). For MLRun, it applies to the jobs created by MLRun. For Nuclio it applies to the pods of the Nuclio-created functions. See Pod priority.
  • You can configure default limits for user jobs for CPU, GPU, and memory. The default limits are set in the MLRun and Nuclio services. The values are applied to the service's user-created pods. You can overwrite the default when creating a job, or a function. See CPU, GPU, and memory limits for user jobs.
  • The dashboard shows the GPU utilization for services and Nuclio functions.
  • When modifying a service in the Service page, the system identifies the related services. You can choose to apply the change to all related services, or only to the service you initially modified.
  • The Services dashboard and the Nuclio page show the GPU utilization.
  • Services (in the Services page) can be filtered by name, type, and running user, and sorted by CPU and memory.
  • You can add Environment variables to a web shell service in the Custom Parameters tab of the service. Variables defined in the service override the global environment variables.

Deployment Specifications

Cloud Deployments

  • Supports upgrading Iguazio version from 3.2.1 & 3.2.2 to v3.4 in managed k8s env: AKS, GKE and EKS.
  • Supports Nvidia A100 for GCP.
  • Supports external docker registries on ACR and GCR. Specify the off-cluster URL when you create the docker registry service in Provazio: In the App Cluster tab, select Use Custom User Docker Registry and type in the URL, the Registry username and the Registry password.
  • Supports preemptible nodes in GKE and Azure AKS.
  • Supports external load balancer for restricted global access mode while allowing node auto scaling groups in GKE.
  • Supports Azure deployment without internet connectivity.
  • EKS can be deployed with an IAM role.

Fixes

  • Fixed issues with Nuclio deployment during node scale out.
  • Improved service deployment upon changes.
  • MPIJobs now run in a GCP Cluster.

Notes

  • Platform start-up, shut-down, and upgrade actions should be coordinated with the Iguazio support team.