{ "cells": [ { "cell_type": "markdown", "source": [ "# Using Versioning in S3 Bucket\n", "\n", "This notebook provides a comprehensive and interactive tutorial on how to effectively use versioning in Amazon S3.\n", "\n", "Reference:\n", "\n", "- [Using Versioning in S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html)\n", "\n", "## 01. Prepare Your Playground\n", "\n", "First, we need to prepare our development environment for a better learning experience. Here are the steps to follow:\n", "\n", "- Create an AWS CLI profile. The profile should have S3 full access and STS get-caller-identity permission (for getting the AWS account id).\n", "- Install the following Python libraries:\n", " - [boto_session_manager](https://pypi.org/project/boto_session_manager/): boto3 session management made easy\n", " - [s3pathlib](https://pypi.org/project/s3pathlib/): s3 manipulation made easy\n", " - [rich](https://pypi.org/project/s3pathlib/): for pretty print" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 160, "outputs": [], "source": [ "# Enter your AWS Profile here\n", "aws_profile = \"awshsh_app_dev_us_east_1\"" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:08.501540Z", "end_time": "2023-04-16T01:24:08.508399Z" } } }, { "cell_type": "code", "execution_count": 161, "outputs": [], "source": [ "# Define a helper function to pretty print boto3 API response\n", "from rich import print as rprint\n", "from pprint import pprint\n", "\n", "def rprint_response(res: dict):\n", " \"\"\"\n", " Pretty print boto3 API response\n", " \"\"\"\n", " if \"ResponseMetadata\" in res:\n", " res.pop(\"ResponseMetadata\")\n", " rprint(res)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:10.672082Z", "end_time": "2023-04-16T01:24:10.676407Z" } } }, { "cell_type": "code", "execution_count": 162, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Try to create the bucket ...\n", "done, bucket is created\n", "Try to turn on bucket versioning ...\n", "done\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\u001B[32m'Status'\u001B[0m: \u001B[32m'Enabled'\u001B[0m\u001B[1m}\u001B[0m\n", "text/html": "
{'Status': 'Enabled'}\n
\n" }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "preview S3 bucket: https://console.aws.amazon.com/s3/buckets/807388292768-us-east-1-learn-s3-versioning?tab=objects\n" ] } ], "source": [ "# Create the test S3 bucket and turn on versioning\n", "from boto_session_manager import BotoSesManager\n", "from s3pathlib import S3Path, context\n", "\n", "bsm = BotoSesManager(profile_name=aws_profile)\n", "context.attach_boto_session(bsm.boto_ses)\n", "\n", "bucket = f\"{bsm.aws_account_id}-{bsm.aws_region}-learn-s3-versioning\"\n", "\n", "# Create the bucket and turn on versioning\n", "def is_bucket_exists() -> bool:\n", " try:\n", " bsm.s3_client.head_bucket(Bucket=bucket)\n", " return True\n", " except bsm.s3_client.exceptions.ClientError as e:\n", " return False\n", "\n", "print(\"Try to create the bucket ...\")\n", "if is_bucket_exists() is False:\n", " kwargs = dict(Bucket=bucket)\n", " if bsm.aws_region != \"us-east-1\":\n", " kwargs[\"CreateBucketConfiguration\"] = dict(LocationConstraint=bsm.aws_region)\n", " bsm.s3_client.create_bucket(**kwargs)\n", " print(\"done, bucket is created\")\n", "else:\n", " print(\"bucket already exists\")\n", "\n", "print(\"Try to turn on bucket versioning ...\")\n", "response = bsm.s3_client.get_bucket_versioning(\n", " Bucket=bucket,\n", ")\n", "if \"Status\" in response: # versioning is already enabled or suspended\n", " pass\n", "else: # versioning is not enabled\n", " bsm.s3_client.put_bucket_versioning(\n", " Bucket=bucket,\n", " VersioningConfiguration=dict(\n", " Status=\"Enabled\",\n", " )\n", " )\n", "print(\"done\")\n", "\n", "# verify if bucket versioning is enabled\n", "response = bsm.s3_client.get_bucket_versioning(\n", " Bucket=bucket,\n", ")\n", "rprint_response(response)\n", "print(f\"preview S3 bucket: {S3Path(bucket).console_url}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:21.479999Z", "end_time": "2023-04-16T01:24:22.107292Z" } } }, { "cell_type": "markdown", "source": [ "## 02. Put and Get\n", "\n", "In this section, we learn the behavior of the put and get API.\n", "\n", "With versioning, everytime you invoke [put_object](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/put_object.html) API, a new version of the object will be created. And everytime you invoke [get_object](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/get_object.html) API, you can get the latest version of the object. You can think of all historical versions ob an object is a last-in-first-out (LIFO) [Stack](https://en.wikipedia.org/wiki/Stack_(abstract_data_type)), the put object is the push operation, and the get object is the pop operation.\n", "\n", "![](./using-versioning-in-s3-buckets-put-get.drawio.svg)\n", "\n", "Let's test this concept by creating a new object in a bucket with versioning turned on. This will be the first version of the object.\n", "\n", "Reference:\n", "\n", "- [Adding object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/AddingObjectstoVersioningEnabledBuckets.html)\n", "- [Retrieving object versions from a versioning-enabled bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RetrievingObjectVersions.html)\n" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 163, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Create a new object, which is also the first version of this object ...\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"99acfefa036b6b3bf0949f1d9ba8acb2\"'\u001B[0m,\n \u001B[32m'ServerSideEncryption'\u001B[0m: \u001B[32m'AES256'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'ETag': '\"99acfefa036b6b3bf0949f1d9ba8acb2\"',\n    'ServerSideEncryption': 'AES256',\n    'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\n}\n
\n" }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "The version id (v1) = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg\n" ] } ], "source": [ "print(\"Create a new object, which is also the first version of this object ...\")\n", "s3path = S3Path(bucket, \"test.txt\")\n", "\n", "res = bsm.s3_client.put_object(\n", " Bucket=s3path.bucket,\n", " Key=s3path.key,\n", " Body=\"content v1\",\n", ")\n", "rprint_response(res)\n", "\n", "v1 = res[\"VersionId\"]\n", "print(f\"The version id (v1) = {v1}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:27.596425Z", "end_time": "2023-04-16T01:24:27.655686Z" } } }, { "cell_type": "markdown", "source": [ "Then we can immediately get the object. By default, the latest version is returned." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 164, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Get the object ...\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'AcceptRanges'\u001B[0m: \u001B[32m'bytes'\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m28\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'ContentLength'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"99acfefa036b6b3bf0949f1d9ba8acb2\"'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\u001B[0m,\n \u001B[32m'ContentType'\u001B[0m: \u001B[32m'binary/octet-stream'\u001B[0m,\n \u001B[32m'ServerSideEncryption'\u001B[0m: \u001B[32m'AES256'\u001B[0m,\n \u001B[32m'Metadata'\u001B[0m: \u001B[1m{\u001B[0m\u001B[1m}\u001B[0m,\n \u001B[32m'Body'\u001B[0m: \u001B[1m<\u001B[0m\u001B[1;95mbotocore.response.StreamingBody\u001B[0m\u001B[39m object at \u001B[0m\u001B[1;36m0x107d61250\u001B[0m\u001B[1m>\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'AcceptRanges': 'bytes',\n    'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()),\n    'ContentLength': 10,\n    'ETag': '\"99acfefa036b6b3bf0949f1d9ba8acb2\"',\n    'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg',\n    'ContentType': 'binary/octet-stream',\n    'ServerSideEncryption': 'AES256',\n    'Metadata': {},\n    'Body': <botocore.response.StreamingBody object at 0x107d61250>\n}\n
\n" }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Content = content v1\n", "The version id (v1) = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg\n" ] } ], "source": [ "print(\"Get the object ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)\n", "rprint_response(res)\n", "\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v1\"\n", "print(\"Content = {}\".format(content))\n", "\n", "v = res[\"VersionId\"]\n", "assert v == v1\n", "print(\"The version id (v1) = {}\".format(v))" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:29.451515Z", "end_time": "2023-04-16T01:24:29.483794Z" } } }, { "cell_type": "markdown", "source": [ "After creating the initial version of the object, we can put a new content to this object to create a new version. Note that you cannot overwrite an existing version because the versioning system is designed to ensure immutability. Therefore, the ``put_object`` API doesn't have an argument called ``VersionId``." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 165, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Put a new version of the object ...\n", "The version id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n" ] } ], "source": [ "print(\"Put a new version of the object ...\")\n", "res = bsm.s3_client.put_object(\n", " Bucket=s3path.bucket,\n", " Key=s3path.key,\n", " Body=\"content v2\",\n", ")\n", "v2 = res[\"VersionId\"]\n", "print(f\"The version id (v2) = {v2}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:30.562574Z", "end_time": "2023-04-16T01:24:30.621625Z" } } }, { "cell_type": "markdown", "source": [ "Now, let's retrieve the object again. By default, the ``get_object`` API fetches the latest version, and we can observe that the version ID has changed from the previous one." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 166, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Get the object again, now it should be the content of v2 ...\n", "Content = content v2\n", "The version id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n" ] } ], "source": [ "print(\"Get the object again, now it should be the content of v2 ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)\n", "\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v2\"\n", "print(f\"Content = {content}\")\n", "\n", "v = res[\"VersionId\"]\n", "assert v == v2\n", "print(f\"The version id (v2) = {v}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:31.341499Z", "end_time": "2023-04-16T01:24:31.377264Z" } } }, { "cell_type": "markdown", "source": [ "We can explicitly get a historical version using version id." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 167, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Explicitly get the v1 version ...\n", "Content = content v1\n", "The version id = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg\n", "As a reference, v1 = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg\n" ] } ], "source": [ "print(\"Explicitly get the v1 version ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=v1)\n", "\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v1\"\n", "print(f\"Content = {content}\")\n", "\n", "v = res[\"VersionId\"]\n", "assert v == v1\n", "print(f\"The version id = {v}\")\n", "print(f\"As a reference, v1 = {v1}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:31.998804Z", "end_time": "2023-04-16T01:24:32.024343Z" } } }, { "cell_type": "markdown", "source": [ "We can also use the [list_object_versions](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/list_object_versions.html) API to list all the historical versions of an object. It will return in order of last modified time, from the latest to the oldest." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 168, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "List all historical versions ...\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'IsTruncated'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'KeyMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'VersionIdMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'Versions'\u001B[0m: \u001B[1m[\u001B[0m\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"96221cd7501efb4f0ce38d99cfb133e5\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;92mTrue\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m31\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"99acfefa036b6b3bf0949f1d9ba8acb2\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m28\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m]\u001B[0m,\n \u001B[32m'Name'\u001B[0m: \u001B[32m'807388292768-us-east-1-learn-s3-versioning'\u001B[0m,\n \u001B[32m'Prefix'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'MaxKeys'\u001B[0m: \u001B[1;36m1000\u001B[0m,\n \u001B[32m'EncodingType'\u001B[0m: \u001B[32m'url'\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'IsTruncated': False,\n    'KeyMarker': '',\n    'VersionIdMarker': '',\n    'Versions': [\n        {\n            'ETag': '\"96221cd7501efb4f0ce38d99cfb133e5\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm',\n            'IsLatest': True,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        },\n        {\n            'ETag': '\"99acfefa036b6b3bf0949f1d9ba8acb2\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        }\n    ],\n    'Name': '807388292768-us-east-1-learn-s3-versioning',\n    'Prefix': 'test.txt',\n    'MaxKeys': 1000,\n    'EncodingType': 'url'\n}\n
\n" }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Number of versions = 2\n" ] } ], "source": [ "print(\"List all historical versions ...\")\n", "res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)\n", "rprint_response(res)\n", "\n", "n_versions = len(res[\"Versions\"])\n", "print(f\"Number of versions = {n_versions}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:32.722590Z", "end_time": "2023-04-16T01:24:32.761558Z" } } }, { "cell_type": "markdown", "source": [ "## 03. Delete\n", "\n", "![](./using-versioning-in-s3-buckets.svg)\n", "\n", "**About Delete**\n", "\n", "There is only one 'delete' API [\n", "delete_object](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/delete_object.html), but there are two ways to use it.\n", "\n", "**1. Without** ``VersionId``\n", "\n", "If you call the ``delete_object`` API without giving the ``VersionId``, then it is a **regular delete**. It will push a \"Deleted\" marker on top of the versions Stack so that the ``get_object`` will find out that the latest version is a \"Deleted\" marker, and then return a 404 (not found) error. However, the content and the historical versions are still there, you can still get them by explicitly giving the ``VersionId``.\n", "\n", "**2. With** ``VersionId``\n", "\n", "If you call the ``delete_object`` API with a ``VersionId``, then it deletes the specific version. This method can also be used to delete a ``Maker``. This method permanently deletes the specific version and the data, it is impossible to recover it.\n", "\n", "**About Marker**\n", "\n", "Most of versioning system with soft-delete feature uses a boolean attribute to indicate whether the version is deleted or not. However AWS S3 versioning implements it differently. The ``Marker`` itself is not an attribute of a version, it is actually a tiny object with a unique ``VersionId`` on top of the latest version. Note that this ``VersionId`` is the identifier of the marker, it is NOT the ``VersionId`` of the object version you deleted.\n", "\n", "Next, we would like to test the deletion behavior." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 169, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Delete the object, it marks the latest version as 'Deleted' ...\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\u001B[32m'DeleteMarker'\u001B[0m: \u001B[3;92mTrue\u001B[0m, \u001B[32m'VersionId'\u001B[0m: \u001B[32m'AiS0gArqcgFLJZuV.XspbK9uzlfENslM'\u001B[0m\u001B[1m}\u001B[0m\n", "text/html": "
{'DeleteMarker': True, 'VersionId': 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM'}\n
\n" }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Marker Id (m2) = AiS0gArqcgFLJZuV.XspbK9uzlfENslM\n", "As a reference, v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n" ] } ], "source": [ "print(\"Delete the object, it marks the latest version as 'Deleted' ...\")\n", "res = bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key)\n", "rprint_response(res)\n", "\n", "m2 = res[\"VersionId\"] # the delete marker of the version 2\n", "print(f\"Marker Id (m2) = {m2}\")\n", "print(f\"As a reference, v2 = {v2}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:33.491551Z", "end_time": "2023-04-16T01:24:33.534056Z" } } }, { "cell_type": "markdown", "source": [ "Then, let's try to get the object. S3 will get the latest version of this object, and find out it is marked as deleted, so it will return a 404 error." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 170, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Get the object, it should returns a 404 error ...\n" ] }, { "ename": "NoSuchKey", "evalue": "An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.", "output_type": "error", "traceback": [ "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m", "\u001B[0;31mNoSuchKey\u001B[0m Traceback (most recent call last)", "Cell \u001B[0;32mIn[170], line 2\u001B[0m\n\u001B[1;32m 1\u001B[0m \u001B[38;5;28mprint\u001B[39m(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mGet the object, it should returns a 404 error ...\u001B[39m\u001B[38;5;124m\"\u001B[39m)\n\u001B[0;32m----> 2\u001B[0m res \u001B[38;5;241m=\u001B[39m \u001B[43mbsm\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43ms3_client\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mget_object\u001B[49m\u001B[43m(\u001B[49m\u001B[43mBucket\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43ms3path\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mbucket\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mKey\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43ms3path\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mkey\u001B[49m\u001B[43m)\u001B[49m\n", "File \u001B[0;32m~/venvs/python/3.8.11/dev_exp_share_venv/lib/python3.8/site-packages/botocore/client.py:530\u001B[0m, in \u001B[0;36mClientCreator._create_api_method.._api_call\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m 526\u001B[0m \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;167;01mTypeError\u001B[39;00m(\n\u001B[1;32m 527\u001B[0m \u001B[38;5;124mf\u001B[39m\u001B[38;5;124m\"\u001B[39m\u001B[38;5;132;01m{\u001B[39;00mpy_operation_name\u001B[38;5;132;01m}\u001B[39;00m\u001B[38;5;124m() only accepts keyword arguments.\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m 528\u001B[0m )\n\u001B[1;32m 529\u001B[0m \u001B[38;5;66;03m# The \"self\" in this scope is referring to the BaseClient.\u001B[39;00m\n\u001B[0;32m--> 530\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_make_api_call\u001B[49m\u001B[43m(\u001B[49m\u001B[43moperation_name\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n", "File \u001B[0;32m~/venvs/python/3.8.11/dev_exp_share_venv/lib/python3.8/site-packages/botocore/client.py:960\u001B[0m, in \u001B[0;36mBaseClient._make_api_call\u001B[0;34m(self, operation_name, api_params)\u001B[0m\n\u001B[1;32m 958\u001B[0m error_code \u001B[38;5;241m=\u001B[39m parsed_response\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mError\u001B[39m\u001B[38;5;124m\"\u001B[39m, {})\u001B[38;5;241m.\u001B[39mget(\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mCode\u001B[39m\u001B[38;5;124m\"\u001B[39m)\n\u001B[1;32m 959\u001B[0m error_class \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mexceptions\u001B[38;5;241m.\u001B[39mfrom_code(error_code)\n\u001B[0;32m--> 960\u001B[0m \u001B[38;5;28;01mraise\u001B[39;00m error_class(parsed_response, operation_name)\n\u001B[1;32m 961\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[1;32m 962\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m parsed_response\n", "\u001B[0;31mNoSuchKey\u001B[0m: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist." ] } ], "source": [ "print(\"Get the object, it should returns a 404 error ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "You can still get the content of the deleted version by explicitly giving the ``VersionId``." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 171, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Directly get the content of a deleted version ...\n", "Content = content v2\n", "Version Id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n" ] } ], "source": [ "print(\"Directly get the content of a deleted version ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=v2)\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v2\"\n", "print(f\"Content = {content}\")\n", "\n", "v = res[\"VersionId\"]\n", "assert v == v2\n", "print(f\"Version Id (v2) = {v}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:35.131974Z", "end_time": "2023-04-16T01:24:35.172037Z" } } }, { "cell_type": "markdown", "source": [ "The ``get_object`` API fails, but the ``list_object_versions`` still working." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 172, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "List all historical versions ...\n", "Now you should see a new field 'DeleteMarkers' and it has a marker object\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'IsTruncated'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'KeyMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'VersionIdMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'Versions'\u001B[0m: \u001B[1m[\u001B[0m\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"96221cd7501efb4f0ce38d99cfb133e5\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m31\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"99acfefa036b6b3bf0949f1d9ba8acb2\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m28\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m]\u001B[0m,\n \u001B[32m'DeleteMarkers'\u001B[0m: \u001B[1m[\u001B[0m\n \u001B[1m{\u001B[0m\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'AiS0gArqcgFLJZuV.XspbK9uzlfENslM'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;92mTrue\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m34\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m]\u001B[0m,\n \u001B[32m'Name'\u001B[0m: \u001B[32m'807388292768-us-east-1-learn-s3-versioning'\u001B[0m,\n \u001B[32m'Prefix'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'MaxKeys'\u001B[0m: \u001B[1;36m1000\u001B[0m,\n \u001B[32m'EncodingType'\u001B[0m: \u001B[32m'url'\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'IsTruncated': False,\n    'KeyMarker': '',\n    'VersionIdMarker': '',\n    'Versions': [\n        {\n            'ETag': '\"96221cd7501efb4f0ce38d99cfb133e5\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        },\n        {\n            'ETag': '\"99acfefa036b6b3bf0949f1d9ba8acb2\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        }\n    ],\n    'DeleteMarkers': [\n        {\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            },\n            'Key': 'test.txt',\n            'VersionId': 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM',\n            'IsLatest': True,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 34, tzinfo=tzutc())\n        }\n    ],\n    'Name': '807388292768-us-east-1-learn-s3-versioning',\n    'Prefix': 'test.txt',\n    'MaxKeys': 1000,\n    'EncodingType': 'url'\n}\n
\n" }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(\"List all historical versions ...\")\n", "print(\"Now you should see a new field 'DeleteMarkers' and it has a marker object\")\n", "res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)\n", "rprint_response(res)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:36.165747Z", "end_time": "2023-04-16T01:24:36.204871Z" } } }, { "cell_type": "markdown", "source": [ "Now we can put a new version on top of the delete marker." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 173, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Put a new version\n", "Version Id (v3) = qTIIVF8nx_NIu0epH3mfveKLml00QBGq\n" ] } ], "source": [ "print(\"Put a new version\")\n", "res = bsm.s3_client.put_object(Bucket=s3path.bucket, Key=s3path.key, Body=\"content v3\")\n", "\n", "v3 = res[\"VersionId\"]\n", "print(f\"Version Id (v3) = {v3}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:37.134996Z", "end_time": "2023-04-16T01:24:37.185239Z" } } }, { "cell_type": "code", "execution_count": 174, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Get the object without giving the version id, it should return the latest version (v3) ...\n", "Content = content v3\n", "Version Id (v3) = qTIIVF8nx_NIu0epH3mfveKLml00QBGq\n" ] } ], "source": [ "print(\"Get the object without giving the version id, it should return the latest version (v3) ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v3\"\n", "print(f\"Content = {content}\")\n", "\n", "v = res[\"VersionId\"]\n", "assert v == v3\n", "print(f\"Version Id (v3) = {v}\")\n" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:37.668144Z", "end_time": "2023-04-16T01:24:37.704767Z" } } }, { "cell_type": "markdown", "source": [ "## 04. Restore\n", "\n", "Firstly, it's important to note that there's no need to \"Restore\" anything as you can still access the data of all historical versions by explicitly giving the VersionId, even if it's marked as \"Deleted\". Typically, when people refer to \"Restore,\" they want to make a deleted historical version the latest version. However, the previous statement is ambiguous and could have two possible meanings:\n", "\n", "1. Copy the content of the deleted historical version to a new version as the latest. To do this, you can retrieve the content of the historical version by specifying the appropriate ``VersionId``, and then create a new version with that content.\n", "2. Delete versions from the top of the version stack until you reach the desired historical version, including the marker that marks the desired historical version as \"Deleted\". To accomplish this, you can list the object versions and markers, sort them by last modified time, and delete the versions and markers from the top of the stack until you reach the desired historical version.\n", "\n", "![](./using-versioning-in-s3-buckets-restore-use-case-1.drawio.svg)\n", "![](./using-versioning-in-s3-buckets-restore-use-case-2.drawio.svg)\n", "\n", "### Copy the content of the deleted historical version to a new version as the latest\n", "\n", "First, let's try the first method. In this example, v2 is the one we deleted. We use the [copy_object](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/copy_object.html) API to copy the content of deleted v2 to a new version as the latest." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 175, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Copy the content of deleted v2 to a new version as the latest ...\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'CopySourceVersionId'\u001B[0m: \u001B[32m'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq'\u001B[0m,\n \u001B[32m'ServerSideEncryption'\u001B[0m: \u001B[32m'AES256'\u001B[0m,\n \u001B[32m'CopyObjectResult'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"96221cd7501efb4f0ce38d99cfb133e5\"'\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m39\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m\n \u001B[1m}\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'CopySourceVersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm',\n    'VersionId': 'Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq',\n    'ServerSideEncryption': 'AES256',\n    'CopyObjectResult': {\n        'ETag': '\"96221cd7501efb4f0ce38d99cfb133e5\"',\n        'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 39, tzinfo=tzutc())\n    }\n}\n
\n" }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "The version id (v4) = Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq\n", "It is different from the v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n" ] } ], "source": [ "print(\"Copy the content of deleted v2 to a new version as the latest ...\")\n", "\n", "res = bsm.s3_client.copy_object(\n", " Bucket=s3path.bucket,\n", " Key=s3path.key,\n", " CopySource=dict(\n", " Bucket=s3path.bucket,\n", " Key=s3path.key,\n", " VersionId=v2,\n", " )\n", ")\n", "rprint_response(res)\n", "\n", "v4 = res[\"VersionId\"]\n", "print(f\"The version id (v4) = {v4}\")\n", "print(f\"It is different from the v2 = {v2}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:38.737420Z", "end_time": "2023-04-16T01:24:38.791565Z" } } }, { "cell_type": "markdown", "source": [ "Then, let's get the object. The content should be the same as v2, but the version id should be different from both v2." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 176, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Get the object, the content should be the same as v2, but the version id should be different from v2 ...\n", "Content = content v2\n", "Version Id (v4) = Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq\n", "As a reference, v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n" ] } ], "source": [ "print(\"Get the object, the content should be the same as v2, but the version id should be different from v2 ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)\n", "\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v2\"\n", "print(f\"Content = {content}\")\n", "\n", "v = res[\"VersionId\"]\n", "assert v == v4\n", "assert v != v2\n", "print(f\"Version Id (v4) = {v4}\")\n", "print(f\"As a reference, v2 = {v2}\")" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:39.762855Z", "end_time": "2023-04-16T01:24:39.791149Z" } } }, { "cell_type": "code", "execution_count": 177, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "List all historical versions ...\n", "Now you should see there are 4 versions of object and 1 delete marker\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'IsTruncated'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'KeyMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'VersionIdMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'Versions'\u001B[0m: \u001B[1m[\u001B[0m\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"96221cd7501efb4f0ce38d99cfb133e5\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;92mTrue\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m39\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"83f457d8c45b9f866c01c7c39ea0d917\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'qTIIVF8nx_NIu0epH3mfveKLml00QBGq'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m38\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"96221cd7501efb4f0ce38d99cfb133e5\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m31\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"99acfefa036b6b3bf0949f1d9ba8acb2\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m28\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m]\u001B[0m,\n \u001B[32m'DeleteMarkers'\u001B[0m: \u001B[1m[\u001B[0m\n \u001B[1m{\u001B[0m\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'AiS0gArqcgFLJZuV.XspbK9uzlfENslM'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m34\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m]\u001B[0m,\n \u001B[32m'Name'\u001B[0m: \u001B[32m'807388292768-us-east-1-learn-s3-versioning'\u001B[0m,\n \u001B[32m'Prefix'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'MaxKeys'\u001B[0m: \u001B[1;36m1000\u001B[0m,\n \u001B[32m'EncodingType'\u001B[0m: \u001B[32m'url'\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'IsTruncated': False,\n    'KeyMarker': '',\n    'VersionIdMarker': '',\n    'Versions': [\n        {\n            'ETag': '\"96221cd7501efb4f0ce38d99cfb133e5\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq',\n            'IsLatest': True,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 39, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        },\n        {\n            'ETag': '\"83f457d8c45b9f866c01c7c39ea0d917\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'qTIIVF8nx_NIu0epH3mfveKLml00QBGq',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 38, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        },\n        {\n            'ETag': '\"96221cd7501efb4f0ce38d99cfb133e5\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        },\n        {\n            'ETag': '\"99acfefa036b6b3bf0949f1d9ba8acb2\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        }\n    ],\n    'DeleteMarkers': [\n        {\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            },\n            'Key': 'test.txt',\n            'VersionId': 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 34, tzinfo=tzutc())\n        }\n    ],\n    'Name': '807388292768-us-east-1-learn-s3-versioning',\n    'Prefix': 'test.txt',\n    'MaxKeys': 1000,\n    'EncodingType': 'url'\n}\n
\n" }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(\"List all historical versions ...\")\n", "print(\"Now you should see there are 4 versions of object and 1 delete marker\")\n", "res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)\n", "rprint_response(res)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:41.200951Z", "end_time": "2023-04-16T01:24:41.247077Z" } } }, { "cell_type": "markdown", "source": [ "### Delete all versions later than the deleted historical version\n", "\n", "Next, we'll explore the second method. First, we'll call the ``list_object_versions`` API to identify all object versions created after v2. We'll also use v2's last update time, which won't change after a version is created due to versioning being turned on, to locate all delete markers created after v2. We'll store all versions to be deleted in a list called ``to_delete``." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 178, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Find all the object versions or delete markers later than v2 ...\n", "need to delete the following versions: ['Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq', 'qTIIVF8nx_NIu0epH3mfveKLml00QBGq', 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM']\n", "they should be [v4, v3, m2]: ['Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq', 'qTIIVF8nx_NIu0epH3mfveKLml00QBGq', 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM']\n", "Then delete all of them ...\n" ] } ], "source": [ "print(\"Find all the object versions or delete markers later than v2 ...\")\n", "\n", "res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)\n", "\n", "to_delete = []\n", "modified_time = None\n", "for dct in res[\"Versions\"]:\n", " if dct[\"VersionId\"] == v2:\n", " modified_time = dct[\"LastModified\"]\n", " break\n", " else:\n", " to_delete.append(dct[\"VersionId\"])\n", "\n", "for dct in res[\"DeleteMarkers\"]:\n", " if dct[\"LastModified\"] > modified_time:\n", " to_delete.append(dct[\"VersionId\"])\n", " else:\n", " break\n", "\n", "print(f\"need to delete the following versions: {to_delete}\")\n", "print(f\"they should be [v4, v3, m2]: {[v4, v3, m2]}\")\n", "\n", "print(\"Then delete all of them ...\")\n", "for id in to_delete:\n", " bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=id)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:46.607064Z", "end_time": "2023-04-16T01:24:46.758279Z" } } }, { "cell_type": "markdown", "source": [ "Now, let's verify. You can see that all versions and markers later than v2 are deleted, we only have v1 and v2 left." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 179, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Get the latest object, it should be the v2 ...\n", "Content = content v2\n", "The version id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n", "As a reference, v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm\n", "List all historical versions ...\n", "Now you should see there are 2 versions of object and no marker\n" ] }, { "data": { "text/plain": "\u001B[1m{\u001B[0m\n \u001B[32m'IsTruncated'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'KeyMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'VersionIdMarker'\u001B[0m: \u001B[32m''\u001B[0m,\n \u001B[32m'Versions'\u001B[0m: \u001B[1m[\u001B[0m\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"96221cd7501efb4f0ce38d99cfb133e5\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;92mTrue\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m31\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m,\n \u001B[1m{\u001B[0m\n \u001B[32m'ETag'\u001B[0m: \u001B[32m'\"99acfefa036b6b3bf0949f1d9ba8acb2\"'\u001B[0m,\n \u001B[32m'Size'\u001B[0m: \u001B[1;36m10\u001B[0m,\n \u001B[32m'StorageClass'\u001B[0m: \u001B[32m'STANDARD'\u001B[0m,\n \u001B[32m'Key'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'VersionId'\u001B[0m: \u001B[32m'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg'\u001B[0m,\n \u001B[32m'IsLatest'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n \u001B[32m'LastModified'\u001B[0m: \u001B[1;35mdatetime.datetime\u001B[0m\u001B[1m(\u001B[0m\u001B[1;36m2023\u001B[0m, \u001B[1;36m4\u001B[0m, \u001B[1;36m16\u001B[0m, \u001B[1;36m5\u001B[0m, \u001B[1;36m24\u001B[0m, \u001B[1;36m28\u001B[0m, \u001B[33mtzinfo\u001B[0m=\u001B[1;35mtzutc\u001B[0m\u001B[1m(\u001B[0m\u001B[1m)\u001B[0m\u001B[1m)\u001B[0m,\n \u001B[32m'Owner'\u001B[0m: \u001B[1m{\u001B[0m\n \u001B[32m'DisplayName'\u001B[0m: \u001B[32m'sanhehu+awshsh-app-dev'\u001B[0m,\n \u001B[32m'ID'\u001B[0m: \u001B[32m'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m}\u001B[0m\n \u001B[1m]\u001B[0m,\n \u001B[32m'Name'\u001B[0m: \u001B[32m'807388292768-us-east-1-learn-s3-versioning'\u001B[0m,\n \u001B[32m'Prefix'\u001B[0m: \u001B[32m'test.txt'\u001B[0m,\n \u001B[32m'MaxKeys'\u001B[0m: \u001B[1;36m1000\u001B[0m,\n \u001B[32m'EncodingType'\u001B[0m: \u001B[32m'url'\u001B[0m\n\u001B[1m}\u001B[0m\n", "text/html": "
{\n    'IsTruncated': False,\n    'KeyMarker': '',\n    'VersionIdMarker': '',\n    'Versions': [\n        {\n            'ETag': '\"96221cd7501efb4f0ce38d99cfb133e5\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm',\n            'IsLatest': True,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        },\n        {\n            'ETag': '\"99acfefa036b6b3bf0949f1d9ba8acb2\"',\n            'Size': 10,\n            'StorageClass': 'STANDARD',\n            'Key': 'test.txt',\n            'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg',\n            'IsLatest': False,\n            'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()),\n            'Owner': {\n                'DisplayName': 'sanhehu+awshsh-app-dev',\n                'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7'\n            }\n        }\n    ],\n    'Name': '807388292768-us-east-1-learn-s3-versioning',\n    'Prefix': 'test.txt',\n    'MaxKeys': 1000,\n    'EncodingType': 'url'\n}\n
\n" }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(\"Get the latest object, it should be the v2 ...\")\n", "res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)\n", "\n", "content = res[\"Body\"].read().decode(\"utf-8\")\n", "assert content == \"content v2\"\n", "print(f\"Content = {content}\")\n", "\n", "v = res[\"VersionId\"]\n", "print(f\"The version id (v2) = {v}\")\n", "print(f\"As a reference, v2 = {v2}\")\n", "\n", "print(\"List all historical versions ...\")\n", "print(\"Now you should see there are 2 versions of object and no marker\")\n", "\n", "res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)\n", "rprint_response(res)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:48.776904Z", "end_time": "2023-04-16T01:24:48.841636Z" } } }, { "cell_type": "markdown", "source": [ "## Clean Up\n", "\n", "We learned the most of the basic operations of using versioning. Now, let's clean up the bucket to avoid cost." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 180, "outputs": [], "source": [ "# Delete all object\n", "s3bucket = S3Path(bucket)\n", "for s3path in s3bucket.iter_objects():\n", " res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)\n", " for dct in res.get(\"Versions\", []):\n", " bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=dct[\"VersionId\"])\n", " for dct in res.get(\"DeleteMarkers\", []):\n", " bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=dct[\"VersionId\"])\n", "\n", "# Delete the bucket\n", "res = bsm.s3_client.delete_bucket(Bucket=bucket)" ], "metadata": { "collapsed": false, "ExecuteTime": { "start_time": "2023-04-16T01:24:59.991364Z", "end_time": "2023-04-16T01:25:01.090744Z" } } }, { "cell_type": "code", "execution_count": null, "outputs": [], "source": [], "metadata": { "collapsed": false } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }