PutRecords

On This Page

Description

Adds records to a stream.

You can optionally assign a record to specific stream shard by specifying a related shard ID (see the Records.ShardId request parameter), or associate the record with a specific partition key to ensure that similar records are assigned to the same shard (see the Records.PartitionKey parameter). By default, the platform assigns records to shards using a Round Robin algorithm. For more information, see Stream Sharding and Partitioning.

Note
The maximum JSON body size for PutRecords requests is 10 MB.

Request

Request Header

Syntax
    POST /<container>/<resource> HTTP/1.1
    Host: <web-APIs URL>
    Content-Type: application/json
    X-v3io-function: PutRecords
    <Authorization OR X-v3io-session-key>: <value>
    
    url = "http://<web-APIs URL>/<container>/<resource>"
    headers = {
                "Content-Type": "application/json",
                "X-v3io-function": "PutRecords",
                "<Authorization OR X-v3io-session-key>": "<value>"
              }
    
    URL Resource Parameters

    The path to the stream to which to add the records. You can optionally set the stream name in the request’s StreamName JSON parameter instead of in the URL.

    Request Data

    Syntax
      {
          "StreamName": "string",
          "Records": [
              {
                  "ClientInfo":   "blob",
                  "Data":         "blob",
                  "PartitionKey": "string",
                  "ShardId":      number
              }
          ]
      }
      
      payload = {
                  "Records": [
                      {
                          "ClientInfo":  "MTIzNA0K",
                          "Data":        "ZGF0YQ0K"
                          }
                      ]
                }
      
      Parameters
      StreamName

      The name of the stream that contains the shard resource.

      • Type: String
      • Requirement: Required if not set in the request URL
      Records

      An array of one or more records to add to the stream, up to a maximum of 1,000 records. If the records limit is exceeded, the request fails.

      • Type: Array of record JSON objects
      • Requirement: Required

      The record JSON object contains these elements:

      ClientInfo

      Custom opaque information that can optionally be provided by the producer. This metadata can be used, for example, to save the data format of a record, or the time at which a sensor or application event was triggered. See Record Metadata.

      • Type: Blob — a Base64 encoded string
      • Maximum Size: 128 bits
      • Requirement: Optional
      Data

      Record data.

      • Type: Blob — a Base64 encoded string
      • Maximum Size: 1 MB
      • Requirement: Required
      PartitionKey

      A partition key with which to associate the record (see Record Metadata). Records with the same partition key are assigned to the same shard, subject to the following exceptions: if a shard ID is also provided for the record (see the Records ShardId request parameter), the record is assigned according to the shard ID, and PartitionKey is ignored. In addition, if you increase a stream’s shard count after its creation (see UpdateStream), new records with a previously used partition key will be assigned either to the same shard that was previously used for this partition key or to a new shard. All records with the same partition key that are added to the stream after the shard-count change will be assigned to the same shard (be it the previously used shard or a new shard). When neither a Shard ID or a partition key is provided in the request, the platform’s default shard-assignment algorithm is used. See also Stream Sharding and Partitioning.

      • Type: String
      • Maximum Size: 256 bytes
      • Requirement: Optional
      ShardId

      The ID of the shard to which to assign the record, as an integer between 0 and one less than the stream’s shard count.
      When both ShardId and PartitionKey are set, the record is assigned according to the shard ID, and PartitionKey is ignored. When neither a Shard ID or a partition key is provided in the request, the platform’s default shard-assignment algorithm is used. See also Stream Sharding and Partitioning.

      • Type: Number
      • Requirement: Optional

      Response

      Response Data

      Syntax
      {
          "FailedRecordCount": number,
          "Records": [
              {
                  "ErrorCode":      number,
                  "ErrorMessage":   "string",
                  "SequenceNumber": number,
                  "ShardId":        number
              }
           ]
      }
      
      Elements
      FailedRecordCount

      The number of records whose submissions failed.

      • Type: Number
      Records

      An array of result objects for each submitted record, in the same order as the records appeared in the submission request.

      • Type: Array of record JSON objects

      For records whose submission succeeded, the record JSON object contains these elements:

      SequenceNumber

      The record’s sequence number, which uniquely identifies the record within the shard. This ID number can be used in a sequence-based Seek operation to get the location of a specific record within a given stream shard.

      • Type: Number
      ShardId

      The ID of the shard to which the record was assigned.

      • Type: Number

      For records whose submission failed, the record JSON object contains these elements:

      ErrorCode

      A unique numeric error code.

      • Type: Number
      ErrorMessage

      An error message in the form of a unique error-code string — see Errors.

      • Type: String

      Errors

      In the event of an error, the response includes a JSON object with an ErrorCode element that contains a unique numeric error code, and an ErrorMessage element that contains one of the following API error messages:
      Error Message Description
      InvalidArgumentException A provided request parameter is not valid for this request.
      Permission denied The sender of the request does not have the required permissions to perform the operation.
      ResourceIsnotStream The specified stream path does not point to a stream.
      ResourceNotFoundException The specified resource does not exist.
      ShardIDOutOfRangeException The specified shard does not exist in this stream.

      Examples

      Add a new record with custom client-information metadata to a MyStream stream. Because the request does not specify a shard ID or a partition key for the record, the record’s shard assignment will be determined by the platform (default):

      Request
        POST /mycontainer/MyDirectory/MyStream/ HTTP/1.1
        Host: https://default-tenant.app.mycluster.iguazio.com:8443
        Content-Type: application/json
        X-v3io-function: PutRecords
        X-v3io-session-key: e8bd4ca2-537b-4175-bf01-8c74963e90bf
        
        {
            "Records": [
                {
                    "ClientInfo": "MTIzNA0K",
                    "Data":       "ZGF0YQ0K"
                }
            ]
        }
        
        import requests
        
        url = "https://default-tenant.app.mycluster.iguazio.com:8443/mycontainer/MyDirectory/MyStream/"
        headers = {
                    "Content-Type": "application/json",
                    "X-v3io-function": "PutRecords",
                    "X-v3io-session-key": "e8bd4ca2-537b-4175-bf01-8c74963e90bf"
                  }
        payload = {
                    "Records": [
                        {
                            "ClientInfo": "MTIzNA0K",
                            "Data": "ZGF0YQ0K"
                            }
                        ]
                  }
        
        response = requests.post(url, json=payload, headers=headers)
        print(response.text)
        
        
        Response
        HTTP/1.1 200 OK
        Content-Type: application/json
        ...
        
        {
            "FailedRecordCount": 0,
            "Records": [
                {
                    "SequenceNumber": 13495,
                    "ShardId": 199
                }
            ]
        }