Software Specifications and Restrictions

On This Page

Following is a list of software specifications and restrictions for version 3.7.2 of the Iguazio AI Platform ("the platform"). For a list of reserved names in the platform, see the reserved-names reference. For a list of known issues for the current release, see the release notes.
(Note: For amount values, M = million; B = billion.)

General

Replication factor 2
Maximum number of concurrent sessions 100,000
Maximum number of rules per data-access policy 100
Maximum number of users 10,000
Maximum number of user groups 64 per user; 1,000 overall.
Note: If a user is a member of more than 64 user groups, only the first 64 groups (which always includes the primary user group) are used for checking authorization.
Usernames

Usernames are subject to the following restrictions:

  • Contain only the following characters:

    • Alphanumeric characters (a–z, A–Z, 0–9)
    • Hyphens (-)
    • Underscores (_)
    • @ is allowed when it is enclosed by text, as in an email format (m@n)

  • Begin with a letter (a–z, A–Z)

  • Length of 1–32 characters
User passwords

Use passwords are subject to the following restrictions:

  • Contain at least one uppercase letter (A–Z)
  • Contain at least one lowercase letter (a–-z)
  • Contain at least one special character (!, @, #, $, %, ^, &, *)
  • Contain at least one digit (0–9)
  • Length of at least 12 characters
Project names

Project names must conform to the following restrictions (RFC 1123 DNS label-name requirements, a.k.a DNS-1123 label):

  • Contain only the following characters:

    • Lowercase letters (a–z) and numeric digits (0–9)
    • Hyphens (-)

  • Begin and end with a lowercase letter (a–z) or a numeric digit (0–9)

  • Length of 1–63 characters

(See the MLRun project-name restrictions for information regarding upgrading old MLRun projects that don't meet these requirements.)
DNS dependency Platform startup requires an accessible Domain Name System (DNS).

Data Elements

Maximum number of objects

The maximum supported number of objects depends on the size of the data-node memory and on the number of data nodes. The Max Objects — Single Node column indicates the maximum number of objects on a single data node (which is also the total maximum number of objects on an Iguazio playground cluster); the Max Objects — Cluster column indicates the maximum number of objects on the entire cluster for a 3-node cluster (Operational Cluster).

Deployment Type Memory Size Max Objects — Single Node Max Objects — Cluster
Cloud — AWS i3.8xlarge 244 GiB 500M 1B
Cloud — AWS i3.4xlarge 122 GiB 180M 360M
Cloud — AWS i3.2xlarge 61 GiB 50M 50M
Cloud — Azure L16s_v2 128 GB 180M 360M
VM — large node 256 GB 500M 1B
VM — small node 128 GB 180M 360M
Maximum number of data containers
  • Cloud — 10
  • VM — 20
Maximum number of collections per cluster

The maximum number of collections (tables/streams/directories) in a cluster depends on the size of the data-node memory.

Deployment Type Memory Size Max Elements
Cloud — AWS i3.8xlarge 244 GiB 10M
Cloud — AWS i3.4xlarge 122 GiB 5M
Cloud — AWS i3.2xlarge 61 GiB 1M
Cloud — Azure L16s_v2 128 GB 5M
VM — large node 256 GB 10M
VM — small node 128 GB 5M
Maximum elements in a collection (table/stream/directory)

The maximum elements in a collection equals the maximum number of objects per data node and depends on the size of the data-node memory.

Deployment Type Memory Size Max Elements
Cloud — AWS i3.8xlarge 244 GiB 500M
Cloud — AWS i3.4xlarge 122 GiB 180M
Cloud — Azure L16s_v2 128 GB 180M
VM — large node 256 GB 500M
VM — small node 128 GB 180M
Maximum object path size The maximum size of the relative path to an object (file) within a given data container is 4,096 bytes.
Maximum object size

The maximum size of an object (file) depends on the interface:

  • For the Simple-Object Web API (PUT Object) — 5 GB

  • For the file-system interface (file creation) — 1 TB.
    The truncate -s command can be used to add a maximum of 240 MB to the current file size.

  • For other APIs — 500 GB.
Maximum NoSQL-table item size The maximum size of a NoSQL-table item (row) — i.e., the maximum total size of the item's attributes — is 1.5 MB.
Displayed object size Displayed object (file) sizes don't include the size of the object's attributes.
Maximum number of distinct attribute names per container 64,000
Maximum number of attributes per object Between 400 and 1600 attributes, depending on the attribute type
Maximum attribute size

The maximum object (item) attribute size is 61,200 bytes for string attributes and 1 MB for other attribute types, subject to the following restrictions:

  • For blob attributes when using the NoSQL Web API, the maximum size applies to the decoded data for the PutItem operation and to the encoded data for the UpdateItem operation.

  • The total size of all attributes whose size is smaller or equal to 200 bytes cannot exceed 8 KB.
Maximum number of shards in a stream 4,096
Maximum stream retention period 1 year
Object names

Names of data objects (such as items and files) are subject to the general file-system naming restrictions, including a maximum length of 255 characters. In addition —
Container names

Container names are subject to the general file-system naming restrictions and the following additional restrictions:

  • Contain only the following characters:

    • Lowercase letters (a–z) and numeric digits (0–9)
    • Hyphens (-)
    • Underscores (_)

  • Begin and end with a lowercase letter (a–z) or a numeric digit (0–9)

  • Contain at least one lowercase letter (a–z)

  • Not contain multiple successive hyphens (-) or underscores (_)

  • Length of 1–128 characters
Attribute names

Attribute names are subject to the general file-system naming restrictions and the following additional restrictions:

  • Contain only the following characters:

    • Alphanumeric characters (a–z, A–Z, 0–9)
    • Underscores (_)

  • Begin either with a letter (a–z, A–Z) or with an underscore (_)

  • Not identical to a reserved name — see Reserved Names

  • Length of 1–255 characters
Sharding-key value The value of an object's sharding key cannot contain periods, because the leftmost period in an object's primary-key value (name) is assumed to be a separator between sharding and sorting keys.
"users" container
  • Don't attempt to delete the predefined "users" container.
  • If you have a web-based shell, or Jupyter Notebook, don't delete the <username> running-user directory in the "users" container or the auto-generated service files in this directory.
Stream-shard file operations Don't perform any file operations other than delete on stream-shard directories by using the Simple-Object Web API or the file-system interface (including commands that trigger such operations, such as vi or cat). Such operations should be executed only via a dedicated streaming API, such as the Streaming Web API.

File System

Note
See also the files and directories information in the Data Elements section.
File and directory names

All files and directories in the platform — including containers, tables, streams, and attributes — are subject to the file-naming restrictions of the Linux operating system.
The size of a file or directory name must be between 1 and 255 characters.

Note
The platform allows using the same name for a file and a directory that reside in the same path. However, you cannot access such files and directories from the file-system interface because of the file-system naming restrictions.
Moving and renaming files The maximum file size when moving or renaming files using a local file-system commands is 30 GB. To move or rename a larger file, use an Hadoop FS command.
Moving (renaming) directories
  • Moving (renaming) of directories is not supported.
  • Renaming of streams is not supported

See also the specific Jupyter Notebook directory copy and move restrictions.
Sparse files Sparse files are not supported: the physical space consumed by such files in the platform is the same as the logical space, including all holes.

Security

Authorization-policies distribution In some cases, authorization-policies distribution can take up to 30 seconds.
Maximum number of imported IdP users 1,000

Expressions

Maximum length of an expression 8 KB
Maximum number of operators in an expression 700
Backslash in expressions The backslash character (\) isn't supported within expressions in the current release.

MLRun

Note
The restrictions refer to MLRun version 1.10.x. For full specifications and restrictions, see the MLRun documentation.
Maximum number of projects
  1. Recommendation: In a playground, up to 50 projects but not restricted.
Project names See the general project-name restrictions.

Nuclio

Note
The restrictions refer to Nuclio version 1.15.21, which is used in version 3.7.2 of the platform. For full specifications and restrictions, see the Nuclio documentation.
Reserved names

The following names are reserved for internal use and cannot be used as the names of Nuclio functions or API gateways:

  • controller
  • dashboard
  • dlx
  • scaler
Project names See the general project-name restrictions.
Offline function deployment

As a general rule, you can deploy Nuclio functions also in air-gapped environments without an internet connection (a.k.a. "dark sites"). However, for code languages (function runtimes) that download files dynamically during the build, offline deployment might either require an additional effort or might not be possible. For example —

  • Offline deployment of Ruby, Node.js, and .NET Core Nuclio functions is currently not supported.

  • Offline deployment of Go Nuclio functions requires implementing a solution in which the necessary module files are available during the build. The reason for this is that platform version 3.7.2 uses Nuclio version 1.15.21, which uses Go modules, which download module files dynamically. You can bypass this problem, for example, by doing either of the following:

    • Use Nuclio to build the function on an environment with an internet connection, and then push the function's Docker image to the Docker Registry service of your air-gapped platform environment. Once the Docker image is available in the system's Docker Registry, you can deploy a Nuclio function using this Docker image.
    • Set up a Go proxy in your offline platform environment that contains the required modules for building the function, and use it to serve the modules for the build. This implementation is outside the scope of the Iguazio support.
Scale to zero

Functions that have been scaled to zero can only be awoken by using an HTTP trigger.

To avoid scaling a function to zero, set its minimum-replicas configuration parameter to a value higher than zero.

Web APIs

NoSQL Web API

Maximum JSON body size 2 MB
Parallel range scan GetItems doesn't support combining parallel table scans and range scans in the same request.

Streaming Web API

Maximum PutRecords JSON body size 10 MB
Maximum record size 2 MB
Maximum number of records that can be added in a single PutRecords operation 1,000
Maximum overall size of records that can be retrieved in a single GetRecords operation 10 MB

Jupyter

Scala notebooks Running Scala code from a Jupyter notebook isn't supported in the current release. Instead, run Scala code by using spark-submit, or use Python.
Copying and moving (renaming) directories
  • The JupyterLab UI handles directory renaming by using the shutil.move command. Because file-system moving (renaming) of directories isn't supported in the platform, the command tries to execute the move by recursively copying the directory to the new location, and then deleting the original directory. To prevent possible data loss as a result of such a recursive copy (which copies only file objects and not additional metadata, such as object attributes), the platform blocks renaming of directories with nested directories in the Jupyter Notebook service. You can still move such directories, when necessary, by running recursive file-system copy and delete commands from a command line. (Jupyter automatically creates a hidden directory for each notebook, therefore every directory with a notebook has a nested directory and cannot be moved from the JupyterLab UI in the platform.) See also the FS directory copy and move restrictions.
  • If a notebook or other file remains open while the parent directory is being moved (renamed), when the move completes you need to re-select the notebook's kernel (such as "Python 3") from the JupyterLab UI before running code from this notebook.
Files and directories deletion from the Jupyter UI To allow deletion of directories (folders) from the Jupyter UI, the trash mechanism is disabled for both files and directories, so deleted items are not moved to the trash and cannot be restored.

V3IO Frames

Maximum write DataFrame size 120 MB
Attribute data types See the github reference for the supported item-attribute data types.
Partitioned tables The NoSQL backend ("nosql"|"kv") doesn't support writing partitioned tables (although you can read partitioned tables with Frames).

Spark

Note
For issues related to running Spark from a Jupyter Notebook, see the Jupyter section.

NoSQL Spark DataFrame

Write-buffer size The default size of write buffers for NoSQL Spark DataFrames is 24 KB. The buffer size is automatically adjusted according to the specific write request but this affects the performance.
Non-string sorting key Range-scan queries on a table with a non-string sorting-key attribute ignore the items' sorting-key values: the query's sharding-key value is still used to identify the relevant data slice more quickly, but the entire slice is scanned rather than scanning only the items within the query's sorting-keys range (as done for a string sorting key). The reason is that the lexicographic sorting-key sort order that's used for storing the items on the data slice might not match the logical sort order for a non-string sorting key. Therefore, for faster range scans, use string sorting keys.

Spark Streaming

Shard-count increase Consuming stream records from new shards after increasing a stream's shard count (using the UpdateStream web-API operation) requires first restarting the Spark Streaming consumer application.

Grafana

Service restart Changes to the Grafana UI — including custom dashboards and data sources — might not be retained when restarting the Grafana service.
Prometheus data is not persisted either for Cluster application services and nor for the Grafana service.

Web Shell

Restricted command shell The web-based shell service doesn't provide a fully functional Linux shell. It's designed mainly for running application services — such as Spark jobs — and for performing basic file-system operations, but it doesn't support all Linux commands and tools.

Dashboard (UI)

Maximum number of concurrent dashboard users 30
Maximum upload file size 2 GB

Backup, Recovery, and High Availability (HA)

Data duplication during system-failure recovery Automated system-failure recovery for ingested stream records, data appended to a simple object, or update expressions might result in duplicate data writes.
Restricted operations in degraded mode

The following restrictions are applicable when the cluster in the degraded mode:

  • Create and delete container operations aren't permitted.

  • The file-system interface doesn't return information about the number of objects in container directories (as reflected in the dashboard's Number of objects directory-metadata field when browsing a data container).
System-attributes backup

Objects' modification-time (__mtime_*) and creation-time (__ctime_*) system attributes aren't restored as part of the platform's backup and upgrade operations.

See Also