Using Versioning in S3 Bucket¶
This notebook provides a comprehensive and interactive tutorial on how to effectively use versioning in Amazon S3.
Reference:
01. Prepare Your Playground¶
First, we need to prepare our development environment for a better learning experience. Here are the steps to follow:
Create an AWS CLI profile. The profile should have S3 full access and STS get-caller-identity permission (for getting the AWS account id).
Install the following Python libraries:
boto_session_manager: boto3 session management made easy
s3pathlib: s3 manipulation made easy
rich: for pretty print
[160]:
# Enter your AWS Profile here
aws_profile = "awshsh_app_dev_us_east_1"
[161]:
# Define a helper function to pretty print boto3 API response
from rich import print as rprint
from pprint import pprint
def rprint_response(res: dict):
"""
Pretty print boto3 API response
"""
if "ResponseMetadata" in res:
res.pop("ResponseMetadata")
rprint(res)
[162]:
# Create the test S3 bucket and turn on versioning
from boto_session_manager import BotoSesManager
from s3pathlib import S3Path, context
bsm = BotoSesManager(profile_name=aws_profile)
context.attach_boto_session(bsm.boto_ses)
bucket = f"{bsm.aws_account_id}-{bsm.aws_region}-learn-s3-versioning"
# Create the bucket and turn on versioning
def is_bucket_exists() -> bool:
try:
bsm.s3_client.head_bucket(Bucket=bucket)
return True
except bsm.s3_client.exceptions.ClientError as e:
return False
print("Try to create the bucket ...")
if is_bucket_exists() is False:
kwargs = dict(Bucket=bucket)
if bsm.aws_region != "us-east-1":
kwargs["CreateBucketConfiguration"] = dict(LocationConstraint=bsm.aws_region)
bsm.s3_client.create_bucket(**kwargs)
print("done, bucket is created")
else:
print("bucket already exists")
print("Try to turn on bucket versioning ...")
response = bsm.s3_client.get_bucket_versioning(
Bucket=bucket,
)
if "Status" in response: # versioning is already enabled or suspended
pass
else: # versioning is not enabled
bsm.s3_client.put_bucket_versioning(
Bucket=bucket,
VersioningConfiguration=dict(
Status="Enabled",
)
)
print("done")
# verify if bucket versioning is enabled
response = bsm.s3_client.get_bucket_versioning(
Bucket=bucket,
)
rprint_response(response)
print(f"preview S3 bucket: {S3Path(bucket).console_url}")
Try to create the bucket ...
done, bucket is created
Try to turn on bucket versioning ...
done
{'Status': 'Enabled'}
preview S3 bucket: https://console.aws.amazon.com/s3/buckets/807388292768-us-east-1-learn-s3-versioning?tab=objects
02. Put and Get¶
In this section, we learn the behavior of the put and get API.
With versioning, everytime you invoke put_object API, a new version of the object will be created. And everytime you invoke get_object API, you can get the latest version of the object. You can think of all historical versions ob an object is a last-in-first-out (LIFO) Stack, the put object is the push operation, and the get object is the pop operation.
Let’s test this concept by creating a new object in a bucket with versioning turned on. This will be the first version of the object.
Reference:
[163]:
print("Create a new object, which is also the first version of this object ...")
s3path = S3Path(bucket, "test.txt")
res = bsm.s3_client.put_object(
Bucket=s3path.bucket,
Key=s3path.key,
Body="content v1",
)
rprint_response(res)
v1 = res["VersionId"]
print(f"The version id (v1) = {v1}")
Create a new object, which is also the first version of this object ...
{ 'ETag': '"99acfefa036b6b3bf0949f1d9ba8acb2"', 'ServerSideEncryption': 'AES256', 'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg' }
The version id (v1) = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg
Then we can immediately get the object. By default, the latest version is returned.
[164]:
print("Get the object ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
rprint_response(res)
content = res["Body"].read().decode("utf-8")
assert content == "content v1"
print("Content = {}".format(content))
v = res["VersionId"]
assert v == v1
print("The version id (v1) = {}".format(v))
Get the object ...
{ 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()), 'ContentLength': 10, 'ETag': '"99acfefa036b6b3bf0949f1d9ba8acb2"', 'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg', 'ContentType': 'binary/octet-stream', 'ServerSideEncryption': 'AES256', 'Metadata': {}, 'Body': <botocore.response.StreamingBody object at 0x107d61250> }
Content = content v1
The version id (v1) = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg
After creating the initial version of the object, we can put a new content to this object to create a new version. Note that you cannot overwrite an existing version because the versioning system is designed to ensure immutability. Therefore, the put_object API doesn’t have an argument called VersionId.
[165]:
print("Put a new version of the object ...")
res = bsm.s3_client.put_object(
Bucket=s3path.bucket,
Key=s3path.key,
Body="content v2",
)
v2 = res["VersionId"]
print(f"The version id (v2) = {v2}")
Put a new version of the object ...
The version id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
Now, let’s retrieve the object again. By default, the get_object API fetches the latest version, and we can observe that the version ID has changed from the previous one.
[166]:
print("Get the object again, now it should be the content of v2 ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
content = res["Body"].read().decode("utf-8")
assert content == "content v2"
print(f"Content = {content}")
v = res["VersionId"]
assert v == v2
print(f"The version id (v2) = {v}")
Get the object again, now it should be the content of v2 ...
Content = content v2
The version id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
We can explicitly get a historical version using version id.
[167]:
print("Explicitly get the v1 version ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=v1)
content = res["Body"].read().decode("utf-8")
assert content == "content v1"
print(f"Content = {content}")
v = res["VersionId"]
assert v == v1
print(f"The version id = {v}")
print(f"As a reference, v1 = {v1}")
Explicitly get the v1 version ...
Content = content v1
The version id = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg
As a reference, v1 = UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg
We can also use the list_object_versions API to list all the historical versions of an object. It will return in order of last modified time, from the latest to the oldest.
[168]:
print("List all historical versions ...")
res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)
rprint_response(res)
n_versions = len(res["Versions"])
print(f"Number of versions = {n_versions}")
List all historical versions ...
{ 'IsTruncated': False, 'KeyMarker': '', 'VersionIdMarker': '', 'Versions': [ { 'ETag': '"96221cd7501efb4f0ce38d99cfb133e5"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm', 'IsLatest': True, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } }, { 'ETag': '"99acfefa036b6b3bf0949f1d9ba8acb2"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } } ], 'Name': '807388292768-us-east-1-learn-s3-versioning', 'Prefix': 'test.txt', 'MaxKeys': 1000, 'EncodingType': 'url' }
Number of versions = 2
03. Delete¶
About Delete
There is only one ‘delete’ API delete_object, but there are two ways to use it.
1. Without VersionId
If you call the delete_object API without giving the VersionId, then it is a regular delete. It will push a “Deleted” marker on top of the versions Stack so that the get_object will find out that the latest version is a “Deleted” marker, and then return a 404 (not found) error. However, the content and the historical versions are still there, you can still get them by explicitly giving the VersionId.
2. With VersionId
If you call the delete_object API with a VersionId, then it deletes the specific version. This method can also be used to delete a Maker. This method permanently deletes the specific version and the data, it is impossible to recover it.
About Marker
Most of versioning system with soft-delete feature uses a boolean attribute to indicate whether the version is deleted or not. However AWS S3 versioning implements it differently. The Marker itself is not an attribute of a version, it is actually a tiny object with a unique VersionId on top of the latest version. Note that this VersionId is the identifier of the marker, it is NOT the VersionId of the object version you deleted.
Next, we would like to test the deletion behavior.
[169]:
print("Delete the object, it marks the latest version as 'Deleted' ...")
res = bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key)
rprint_response(res)
m2 = res["VersionId"] # the delete marker of the version 2
print(f"Marker Id (m2) = {m2}")
print(f"As a reference, v2 = {v2}")
Delete the object, it marks the latest version as 'Deleted' ...
{'DeleteMarker': True, 'VersionId': 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM'}
Marker Id (m2) = AiS0gArqcgFLJZuV.XspbK9uzlfENslM
As a reference, v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
Then, let’s try to get the object. S3 will get the latest version of this object, and find out it is marked as deleted, so it will return a 404 error.
[170]:
print("Get the object, it should returns a 404 error ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
Get the object, it should returns a 404 error ...
---------------------------------------------------------------------------
NoSuchKey Traceback (most recent call last)
Cell In[170], line 2
1 print("Get the object, it should returns a 404 error ...")
----> 2 res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
File ~/venvs/python/3.8.11/dev_exp_share_venv/lib/python3.8/site-packages/botocore/client.py:530, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
526 raise TypeError(
527 f"{py_operation_name}() only accepts keyword arguments."
528 )
529 # The "self" in this scope is referring to the BaseClient.
--> 530 return self._make_api_call(operation_name, kwargs)
File ~/venvs/python/3.8.11/dev_exp_share_venv/lib/python3.8/site-packages/botocore/client.py:960, in BaseClient._make_api_call(self, operation_name, api_params)
958 error_code = parsed_response.get("Error", {}).get("Code")
959 error_class = self.exceptions.from_code(error_code)
--> 960 raise error_class(parsed_response, operation_name)
961 else:
962 return parsed_response
NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
You can still get the content of the deleted version by explicitly giving the VersionId.
[171]:
print("Directly get the content of a deleted version ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=v2)
content = res["Body"].read().decode("utf-8")
assert content == "content v2"
print(f"Content = {content}")
v = res["VersionId"]
assert v == v2
print(f"Version Id (v2) = {v}")
Directly get the content of a deleted version ...
Content = content v2
Version Id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
The get_object API fails, but the list_object_versions still working.
[172]:
print("List all historical versions ...")
print("Now you should see a new field 'DeleteMarkers' and it has a marker object")
res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)
rprint_response(res)
List all historical versions ...
Now you should see a new field 'DeleteMarkers' and it has a marker object
{ 'IsTruncated': False, 'KeyMarker': '', 'VersionIdMarker': '', 'Versions': [ { 'ETag': '"96221cd7501efb4f0ce38d99cfb133e5"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } }, { 'ETag': '"99acfefa036b6b3bf0949f1d9ba8acb2"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } } ], 'DeleteMarkers': [ { 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' }, 'Key': 'test.txt', 'VersionId': 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM', 'IsLatest': True, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 34, tzinfo=tzutc()) } ], 'Name': '807388292768-us-east-1-learn-s3-versioning', 'Prefix': 'test.txt', 'MaxKeys': 1000, 'EncodingType': 'url' }
Now we can put a new version on top of the delete marker.
[173]:
print("Put a new version")
res = bsm.s3_client.put_object(Bucket=s3path.bucket, Key=s3path.key, Body="content v3")
v3 = res["VersionId"]
print(f"Version Id (v3) = {v3}")
Put a new version
Version Id (v3) = qTIIVF8nx_NIu0epH3mfveKLml00QBGq
[174]:
print("Get the object without giving the version id, it should return the latest version (v3) ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
content = res["Body"].read().decode("utf-8")
assert content == "content v3"
print(f"Content = {content}")
v = res["VersionId"]
assert v == v3
print(f"Version Id (v3) = {v}")
Get the object without giving the version id, it should return the latest version (v3) ...
Content = content v3
Version Id (v3) = qTIIVF8nx_NIu0epH3mfveKLml00QBGq
04. Restore¶
Firstly, it’s important to note that there’s no need to “Restore” anything as you can still access the data of all historical versions by explicitly giving the VersionId, even if it’s marked as “Deleted”. Typically, when people refer to “Restore,” they want to make a deleted historical version the latest version. However, the previous statement is ambiguous and could have two possible meanings:
Copy the content of the deleted historical version to a new version as the latest. To do this, you can retrieve the content of the historical version by specifying the appropriate
VersionId, and then create a new version with that content.Delete versions from the top of the version stack until you reach the desired historical version, including the marker that marks the desired historical version as “Deleted”. To accomplish this, you can list the object versions and markers, sort them by last modified time, and delete the versions and markers from the top of the stack until you reach the desired historical version.
Copy the content of the deleted historical version to a new version as the latest¶
First, let’s try the first method. In this example, v2 is the one we deleted. We use the copy_object API to copy the content of deleted v2 to a new version as the latest.
[175]:
print("Copy the content of deleted v2 to a new version as the latest ...")
res = bsm.s3_client.copy_object(
Bucket=s3path.bucket,
Key=s3path.key,
CopySource=dict(
Bucket=s3path.bucket,
Key=s3path.key,
VersionId=v2,
)
)
rprint_response(res)
v4 = res["VersionId"]
print(f"The version id (v4) = {v4}")
print(f"It is different from the v2 = {v2}")
Copy the content of deleted v2 to a new version as the latest ...
{ 'CopySourceVersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm', 'VersionId': 'Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq', 'ServerSideEncryption': 'AES256', 'CopyObjectResult': { 'ETag': '"96221cd7501efb4f0ce38d99cfb133e5"', 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 39, tzinfo=tzutc()) } }
The version id (v4) = Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq
It is different from the v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
Then, let’s get the object. The content should be the same as v2, but the version id should be different from both v2.
[176]:
print("Get the object, the content should be the same as v2, but the version id should be different from v2 ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
content = res["Body"].read().decode("utf-8")
assert content == "content v2"
print(f"Content = {content}")
v = res["VersionId"]
assert v == v4
assert v != v2
print(f"Version Id (v4) = {v4}")
print(f"As a reference, v2 = {v2}")
Get the object, the content should be the same as v2, but the version id should be different from v2 ...
Content = content v2
Version Id (v4) = Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq
As a reference, v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
[177]:
print("List all historical versions ...")
print("Now you should see there are 4 versions of object and 1 delete marker")
res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)
rprint_response(res)
List all historical versions ...
Now you should see there are 4 versions of object and 1 delete marker
{ 'IsTruncated': False, 'KeyMarker': '', 'VersionIdMarker': '', 'Versions': [ { 'ETag': '"96221cd7501efb4f0ce38d99cfb133e5"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq', 'IsLatest': True, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 39, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } }, { 'ETag': '"83f457d8c45b9f866c01c7c39ea0d917"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'qTIIVF8nx_NIu0epH3mfveKLml00QBGq', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 38, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } }, { 'ETag': '"96221cd7501efb4f0ce38d99cfb133e5"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } }, { 'ETag': '"99acfefa036b6b3bf0949f1d9ba8acb2"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } } ], 'DeleteMarkers': [ { 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' }, 'Key': 'test.txt', 'VersionId': 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 34, tzinfo=tzutc()) } ], 'Name': '807388292768-us-east-1-learn-s3-versioning', 'Prefix': 'test.txt', 'MaxKeys': 1000, 'EncodingType': 'url' }
Delete all versions later than the deleted historical version¶
Next, we’ll explore the second method. First, we’ll call the list_object_versions API to identify all object versions created after v2. We’ll also use v2’s last update time, which won’t change after a version is created due to versioning being turned on, to locate all delete markers created after v2. We’ll store all versions to be deleted in a list called to_delete.
[178]:
print("Find all the object versions or delete markers later than v2 ...")
res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)
to_delete = []
modified_time = None
for dct in res["Versions"]:
if dct["VersionId"] == v2:
modified_time = dct["LastModified"]
break
else:
to_delete.append(dct["VersionId"])
for dct in res["DeleteMarkers"]:
if dct["LastModified"] > modified_time:
to_delete.append(dct["VersionId"])
else:
break
print(f"need to delete the following versions: {to_delete}")
print(f"they should be [v4, v3, m2]: {[v4, v3, m2]}")
print("Then delete all of them ...")
for id in to_delete:
bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=id)
Find all the object versions or delete markers later than v2 ...
need to delete the following versions: ['Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq', 'qTIIVF8nx_NIu0epH3mfveKLml00QBGq', 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM']
they should be [v4, v3, m2]: ['Ylg.6BFISfzYjAnxabwCwHAzA.fkgjBq', 'qTIIVF8nx_NIu0epH3mfveKLml00QBGq', 'AiS0gArqcgFLJZuV.XspbK9uzlfENslM']
Then delete all of them ...
Now, let’s verify. You can see that all versions and markers later than v2 are deleted, we only have v1 and v2 left.
[179]:
print("Get the latest object, it should be the v2 ...")
res = bsm.s3_client.get_object(Bucket=s3path.bucket, Key=s3path.key)
content = res["Body"].read().decode("utf-8")
assert content == "content v2"
print(f"Content = {content}")
v = res["VersionId"]
print(f"The version id (v2) = {v}")
print(f"As a reference, v2 = {v2}")
print("List all historical versions ...")
print("Now you should see there are 2 versions of object and no marker")
res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)
rprint_response(res)
Get the latest object, it should be the v2 ...
Content = content v2
The version id (v2) = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
As a reference, v2 = QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm
List all historical versions ...
Now you should see there are 2 versions of object and no marker
{ 'IsTruncated': False, 'KeyMarker': '', 'VersionIdMarker': '', 'Versions': [ { 'ETag': '"96221cd7501efb4f0ce38d99cfb133e5"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'QrXyu1i9PonpwqmYM8U_6eeDMxoLBqjm', 'IsLatest': True, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 31, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } }, { 'ETag': '"99acfefa036b6b3bf0949f1d9ba8acb2"', 'Size': 10, 'StorageClass': 'STANDARD', 'Key': 'test.txt', 'VersionId': 'UkxPmCOzhwTtzSRnq.h8DsoRXd1A9fLg', 'IsLatest': False, 'LastModified': datetime.datetime(2023, 4, 16, 5, 24, 28, tzinfo=tzutc()), 'Owner': { 'DisplayName': 'sanhehu+awshsh-app-dev', 'ID': 'd8468a5d68b2a6e29f0436ec9b64ed212e2b3a272d15429fbfc58a3c35ff4bf7' } } ], 'Name': '807388292768-us-east-1-learn-s3-versioning', 'Prefix': 'test.txt', 'MaxKeys': 1000, 'EncodingType': 'url' }
Clean Up¶
We learned the most of the basic operations of using versioning. Now, let’s clean up the bucket to avoid cost.
[180]:
# Delete all object
s3bucket = S3Path(bucket)
for s3path in s3bucket.iter_objects():
res = bsm.s3_client.list_object_versions(Bucket=s3path.bucket, Prefix=s3path.key)
for dct in res.get("Versions", []):
bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=dct["VersionId"])
for dct in res.get("DeleteMarkers", []):
bsm.s3_client.delete_object(Bucket=s3path.bucket, Key=s3path.key, VersionId=dct["VersionId"])
# Delete the bucket
res = bsm.s3_client.delete_bucket(Bucket=bucket)
[ ]: