s3cmd (Simple Storage Service Command Line Tool and API)

s3cmd is a command-line tool for working with S3-compatible object storage. It lets you create buckets, upload and download files, and manage objects using standard S3 operations — all from your terminal or scripts.

American Cloud A2 Object Storage is S3-compatible, so s3cmd works against it once you point it at the correct endpoint and enter your keys. Follow the steps below to install and configure it.

Install

On Debian/Ubuntu:

sudo apt install s3cmd

On macOS (Homebrew):

brew install s3cmd

Configure s3cmd

s3cmd needs three things: your Object Storage access key, your secret key, and the endpoint for your region.

Get your keys and endpoint

Sign in to the American Cloud CMP.
In the left navigation, choose Storage.
On the A2 Object Storage tab, select Manage.
Open Object Storage Settings, then select Keys to view your access key and secret key.

Your endpoint is the regional host shown in the CMP. For the West region it is a2-west.americancloud.com. Use the endpoint for the region your bucket lives in.

Recommended: create the config file directly

The quickest and most reliable way to set up s3cmd for American Cloud is to write the config file yourself — it avoids a wizard prompt that's easy to get wrong (see the note below).

Create ~/.s3cfg with the following contents, replacing the two key values with your own:

[default]
access_key = YOUR_ACCESS_KEY
secret_key = YOUR_SECRET_KEY
host_base = a2-west.americancloud.com
host_bucket = a2-west.americancloud.com
use_https = True

That's everything s3cmd needs. Note that host_base and host_bucket are set to the same value — this tells s3cmd to use path-style addressing (https://a2-west.americancloud.com/your-bucket), which is what A2 Object Storage expects.

Verify it works by listing your buckets:

s3cmd ls

If your keys and endpoint are correct, this returns your buckets (or nothing at all, if you haven't created any yet) with no error.

Alternative: the `s3cmd --configure` wizard

You can also configure interactively, but one prompt is easy to get wrong — read the note before running it. Start the wizard with:

s3cmd --configure

Enter the following values, pressing Enter to accept the default for anything not listed:

Prompt	Enter
Access Key	your access key
Secret Key	your secret key
Default Region [US]	leave default (press Enter)
S3 Endpoint	`a2-west.americancloud.com`
DNS-style bucket+hostname:port template	`a2-west.americancloud.com`
Encryption password	leave blank
Path to GPG program	leave default
Use HTTPS protocol [Yes]	leave default
HTTP Proxy server name	leave blank

When prompted, choose Y to run the connection test, then Y to save. The wizard writes your settings to ~/.s3cfg.

Editing the config later

To rotate keys or change your endpoint at any time, edit ~/.s3cfg directly:

nano ~/.s3cfg

Make sure both host_base and host_bucket point at your regional endpoint (e.g. a2-west.americancloud.com) and that your access_key and secret_key are current. A correct file looks like this:

[default]
access_key = YOUR_ACCESS_KEY
secret_key = YOUR_SECRET_KEY
host_base = a2-west.americancloud.com
host_bucket = a2-west.americancloud.com
use_https = True

Add Buckets

Object storage buckets are containers for storing and organizing large volumes of unstructured data, such as files, images, and videos, in the cloud. They provide scalable, durable, and cost-effective storage solutions, allowing users to upload, retrieve, and manage data using APIs or web interfaces. Below are list of commands for adding buckets.

Make Bucket Command

# Use mb (make bucket) command
s3cmd mb s3://americancloud-1

cloud@Compute-1:~$ s3cmd mb s3://americancloud-1
Bucket 's3://americancloud-1/' created

Using the ls command list buckets.

s3cmd ls

The new bucket is listed.

cloud@Compute-1:~$ s3cmd ls
2023-04-19 23:46  s3://americancloud-1
2023-04-19 21:02  s3://bucketac2

As expected the bucket has been placed inside AC CMP.

Removing a Bucket

Removing buckets is a process that permanently deletes a bucket and all its objects. To remove a bucket, the user must have appropriate permissions, and all objects within the bucket must be deleted first. Once a bucket is removed, its data cannot be recovered. It is important to exercise caution and ensure backups are in place before deleting any buckets in S3.

Remove Bucket Command

s3cmd rb

cloud@Compute-1:~$ s3cmd rb s3://ac-123
Bucket 's3://ac-123/' removed

CMP side the bucket has been removed as well.

List Buckets and Files

Listing files in S3 involves retrieving a list of objects (files) stored within a specific bucket. The list typically includes information such as object names, sizes, and metadata. It can be useful for navigating and managing objects in S3, including copying, deleting, or downloading files. Proper access permissions and authentication are required to list files in S3, ensuring data security and privacy.

List command

The ls command will list the buckets within Object Storage.

s3cmd ls

cloud@Compute-1:~$ s3cmd ls
2023-04-19 23:46  s3://americancloud-1
2023-04-20 17:51  s3://americancloud-2

List files within a bucket by running the ls s3://*bucketname.

s3cmd ls s3://americancloud-1

cloud@Compute-1:~$ s3cmd ls s3://americancloud-1
2023-04-20 03:16 89.6904296875k  s3://americancloud-1/AC is Awesome.pages
2023-04-20 00:39 148.197265625k  s3://americancloud-1/Screenshot 2023-04-19 at 6.52.20 PM-20230420123958.png
2023-04-20 00:20 148.90234375k  s3://americancloud-1/Screenshot 2023-04-19 at 7.19.54 PM-20230420122021.png
2023-04-20 17:15         0   s3://americancloud-1/americancloudisawesome.txt
2023-04-20 17:14         0   s3://americancloud-1/sample.txt

Additionally, list all files within all buckets by executing s3cmd la.

s3cmd la

cloud@Compute-1:~$ s3cmd la
2023-04-20 03:16 89.6904296875k  s3://americancloud-1/AC is Awesome.pages
2023-04-20 00:39 148.197265625k  s3://americancloud-1/Screenshot 2023-04-19 at 6.52.20 PM-20230420123958.png
2023-04-20 00:20 148.90234375k  s3://americancloud-1/Screenshot 2023-04-19 at 7.19.54 PM-20230420122021.png
2023-04-20 17:15         0   s3://americancloud-1/americancloudisawesome.txt
2023-04-20 17:14         0   s3://americancloud-1/sample.txt

2023-04-20 17:52         0   s3://americancloud-2/americancloudisawesome.txt

Add Files

The "put" command in S3 is a command-line operation that allows users to upload (put) objects (files) from their local system to an S3 bucket. The "put" command requires specifying the source file path, destination S3 bucket name, and object key (file name) to store the object in S3. Proper permissions and authentication are necessary for successful object uploads.

PUT Command

Single file upload

s3cmd put /file s3://americancloud-1

s3cmd put americancloudisawesome.txt s3://americancloud-1
upload: 'americancloudisawesome.txt' -> 's3://americancloud-1/americancloudisawesome.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

Multiple file upload

s3cmd put ac1.txt ac2.txt path/to/ac3.txt s3://americancloud-1

s3cmd put acisawesome.txt americancloudisawesome.txt s3://bucketac4
upload: 'acisawesome.txt' -> 's3://bucketac4/acisawesome.txt'  [1 of 2]
 0 of 0     0% in    0s     0.00 B/s  done
upload: 'americancloudisawesome.txt' -> 's3://bucketac4/americancloudisawesome.txt'  [2 of 2]
 0 of 0     0% in    0s     0.00 B/s  done

Change name during upload

s3cmd put ac1.txt s3://americancloud-1/newname.txt

s3cmd put test.txt s3://bucketac4/ac-2.txt
upload: 'test.txt' -> 's3://bucketac4/ac-2.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done

If desired an entire director can be moved using 'sync' command. Idea for backup scenarios

cloud@Compute-AC-9:~$ s3cmd sync /home/cloud s3://bucketac4
upload: '/home/cloud/.bash_history' -> 's3://bucketac4/cloud/.bash_history'  [1 of 12]
 0 of 0     0% in    0s     0.00 B/s  done
upload: '/home/cloud/.bash_logout' -> 's3://bucketac4/cloud/.bash_logout'  [2 of 12]
 220 of 220   100% in    0s     7.99 KB/s  done
upload: '/home/cloud/.bashrc' -> 's3://bucketac4/cloud/.bashrc'  [3 of 12]

Retrieving Files

To retrieve files in S3, a cloud-based object storage service, you can use the S3 API or S3 console. First, authenticate and authorize access, then specify the S3 bucket and object key to identify the file. Use the appropriate method, such as GET, to retrieve the file from S3. Optionally, you can configure access control and encryption settings for added security.

GET Command

Single file download

s3cmd get s3://[bucketname]/filename

s3cmd get s3://bucketac4/ac-2.txt
download: 's3://bucketac4/ac-2.txt' -> './ac-2.txt'  [1 of 1]
0 of 0     0% in    0s     0.00 B/s  done

Multiple file download

s3cmd get s3://bucketac4/test1.txt s3://bucketac4/test2.txt
download: 's3://bucketac4/test1.txt' -> './test1.txt'  [1 of 2]
0 of 0     0% in    0s     0.00 B/s  done
download: 's3://bucketac4/test2.txt' -> './test2.txt'  [2 of 2]
0 of 0     0% in    0s     0.00 B/s  done

Change file name

s3cmd get s3://[bucketname]/filename newfilename

s3cmd get s3://bucketac4/ac-4.txt ac-5.txt --recursive
download: 's3://bucketac4/ac-4.txt' -> 'ac-5.txt'  [1 of 1]
0 of 0     0% in    0s     0.00 B/s  done

Use of --recursive. To pull all files from a bucket use the recursive flag.

s3cmd get s3://[bucketname]/ --recursive

s3cmd get s3://bucketac4/ --recursive
download: 's3://bucketac4/Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png' -> './Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png'  [1 of 10]
512226 of 512226   100% in    0s  1229.82 KB/s  done
download: 's3://bucketac4/Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png' -> './Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png'  [2 of 10]
42789 of 42789   100% in    0s   432.86 KB/s  done

s3cmd rm s3://[bucketname]/filename

s3cmd rm s3://bucketac4/ac-5.txt
delete: 's3://bucketac4/ac-5.txt'

Remove multiple files

s3cmd rm s3://bucketac4/ac-2.txt s3://bucketac4/ac-4.txt
delete: 's3://bucketac4/ac-2.txt'
delete: 's3://bucketac4/ac-4.txt'

Remove all files from a bucket use the recursive and force flag.

s3cmd rm s3://[bucketname]/ --recursive --force

cloud@Compute-1:~$ s3cmd rm s3://bucketac4/ --recursive --force
delete: 's3://bucketac4/Screenshot 2023-04-18 at 11.03.46 PM-20230423120714.png'
delete: 's3://bucketac4/Screenshot 2023-04-21 at 5.20.19 PM-20230423120721.png'