Mountpoint for Amazon S3 -Build & Demo

On AWS Pi Day 2023, Amazon announced Mountpoint for Amazon S3, an open-source simple file client for mounting S3 buckets as a local file system. It is optimized for read-heavy workloads that require high throughput and is mainly focused on high-performance access to large data sets. So naturally, it does not offer rich POSIX semantics. If your use-case demands POSIX semantics, then use Amazon EFS or Amazon FSx.

It is currently an alpha release and not ready for production yet. It doesn’t support writes now, and in future, only sequential writes will be supported.

Here, I'll use Ubuntu distribution of WSL2 (Windows Subsystem for Linux) running on Windows 11 for the demo.

abhijit@AwsJunkie:~$ uname -a
Linux AwsJunkie 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build Mountpoint for Amazon S3

At the time of writing of his article, installer is not available for the alpha release of Mountpoint for Amazon S3. Therefore, we'll build it from source.

Mountpoint for Amazon S3 is built using Rust programming language. So, we'll install few dependencies and Rust compiler.

Update current packages and upgrade as a best practice.

abhijit@AwsJunkie:~$ sudo apt update -y && sudo apt upgrade -y

Install dependencies.

abhijit@AwsJunkie:~$ sudo apt install clang cmake build-essential pkg-config fuse libfuse-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  libntfs-3g89
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  binfmt-support bzip2 clang-14 cmake-data cpp cpp-11 dh-elpa-helper dpkg-dev emacsen-common fakeroot
  fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base icu-devtools lib32gcc-s1 lib32stdc++6:
  linux-libc-dev llvm-14 llvm-14-dev llvm-14-linker-tools llvm-14-runtime llvm-14-tools lto-disabled-list
  make manpages-dev pkg-config python3-pygments rpcsvc-proto
0 upgraded, 98 newly installed, 2 to remove and 0 not upgraded.
Need to get 200 MB of archives.
After this operation, 958 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 binfmt-support amd64 2.2.1-2 [55.8 kB]
:
Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-devtools amd64 2.35-0ubuntu3.1 [28.9 kB]
Setting up cmake (3.22.1-1ubuntu1.22.04.1) ...
Setting up libc6-dev:amd64 (2.35-0ubuntu3.1) ...
:
Setting up g++ (4:11.2.0-1ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up build-essential (12.9ubuntu3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for install-info (6.8-4build1) ...

Install Rust compiler using rustup. For WSL (Windows Subsystem for Linux), execute curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh and follow on-screen instructions.

abhijit@AwsJunkie:~$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
info: downloading installer

Welcome to Rust!

This will download and install the official compiler for the Rust
programming language, and its package manager, Cargo.

Rustup metadata and toolchains will be installed into the Rustup
home directory, located at:

  /home/abhijit/.rustup

This can be modified with the RUSTUP_HOME environment variable.

The Cargo home directory is located at:

  /home/abhijit/.cargo

This can be modified with the CARGO_HOME environment variable.

The cargo, rustc, rustup and other commands will be added to
Cargo's bin directory, located at:

  /home/abhijit/.cargo/bin

This path will then be added to your PATH environment variable by
modifying the profile files located at:

  /home/abhijit/.profile
  /home/abhijit/.bashrc

You can uninstall at any time with rustup self uninstall and
these changes will be reverted.

Current installation options:


   default host triple: x86_64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with installation (default)
2) Customize installation
3) Cancel installation
>1

info: profile set to 'default'
info: default host triple is x86_64-unknown-linux-gnu
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: downloading component 'cargo'
  6.7 MiB /   6.7 MiB (100 %)   3.2 MiB/s in  1s ETA:  0s
:
 67.6 MiB /  67.6 MiB (100 %)  15.0 MiB/s in  4s ETA:  0s
info: installing component 'rustfmt'
info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'

  stable-x86_64-unknown-linux-gnu installed - rustc 1.68.0 (2c8cc3432 2023-03-06)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source "$HOME/.cargo/env"

If everything goes well, we'll see "Rust is installed now. Great!" message. Reload Cargo's bin directory ( $HOME/.cargo/bin) to current shell.

abhijit@AwsJunkie:~$ source "$HOME/.cargo/env"

To verify, check version.

abhijit@AwsJunkie:~$ rustc --version
rustc 1.68.0 (2c8cc3432 2023-03-06)

Clone the mountpoint-s3 github repository.

abhijit@AwsJunkie:~$ git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git
Cloning into 'mountpoint-s3'...
remote: Enumerating objects: 7467, done.
remote: Counting objects: 100% (7467/7467), done.
remote: Compressing objects: 100% (2488/2488), done.
remote: Total 7467 (delta 4560), reused 7364 (delta 4511), pack-reused 0
Receiving objects: 100% (7467/7467), 1.91 MiB | 7.90 MiB/s, done.
Resolving deltas: 100% (4560/4560), done.
Submodule 'aws-c-auth' (https://github.com/awslabs/aws-c-auth.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-auth'
Submodule 'aws-c-cal' (https://github.com/awslabs/aws-c-cal.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-cal'
Submodule 'aws-c-common' (https://github.com/awslabs/aws-c-common.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-common'
Submodule 'aws-c-compression' (https://github.com/awslabs/aws-c-compression.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-compression'
Submodule 'aws-c-http' (https://github.com/awslabs/aws-c-http.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-http'
Submodule 'aws-c-io' (https://github.com/awslabs/aws-c-io.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-io'
Submodule 'aws-c-s3' (https://github.com/awslabs/aws-c-s3.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-s3'
Submodule 'aws-c-sdkutils' (https://github.com/awslabs/aws-c-sdkutils.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-c-sdkutils'
Submodule 'aws-checksums' (https://github.com/awslabs/aws-checksums.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-checksums'
Submodule 'aws-lc' (https://github.com/awslabs/aws-lc.git) registered for path 'mountpoint-s3-crt-sys/crt/aws-lc'
Submodule 's2n-tls' (https://github.com/aws/s2n-tls.git) registered for path 'mountpoint-s3-crt-sys/crt/s2n-tls'
Cloning into '/home/abhijit/mountpoint-s3/mountpoint-s3-crt-sys/crt/aws-c-auth'...
:
:
remote: Total 55647 (delta 75), reused 90 (delta 54), pack-reused 55518
Receiving objects: 100% (55647/55647), 26.13 MiB | 11.83 MiB/s, done.
Resolving deltas: 100% (39677/39677), done.
:
Submodule 'tests/cbmc/aws-verification-model-for-libcrypto' (https://github.com/awslabs/aws-verification-model-for-libcrypto.git) registered for path 'mountpoint-s3-crt-sys/crt/s2n-tls/tests/cbmc/aws-verification-model-for-libcrypto'
Cloning into '/home/abhijit/mountpoint-s3/mountpoint-s3-crt-sys/crt/s2n-tls/tests/cbmc/aws-verification-model-for-libcrypto'...
remote: Enumerating objects: 248, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 248 (delta 99), reused 63 (delta 53), pack-reused 108
Receiving objects: 100% (248/248), 111.55 KiB | 1.49 MiB/s, done.
Resolving deltas: 100% (153/153), done.
Submodule path 'mountpoint-s3-crt-sys/crt/s2n-tls/tests/cbmc/aws-verification-model-for-libcrypto': checked out '82f4f0989a98752050f58c74a89942502a323ca7'

Got inside the cloned project (mountpoint-s3) and create release build using Cargo - Rust package manager.

abhijit@AwsJunkie:~$ ls
mountpoint-s3
abhijit@AwsJunkie:~$ cd mountpoint-s3/
abhijit@AwsJunkie:~/mountpoint-s3$
abhijit@AwsJunkie:~/mountpoint-s3$ cargo build --release
  Updating crates.io index
  Downloaded proc-macro2 v1.0.51
  :
  Downloaded static_assertions v1.1.0
  Downloaded 144 crates (13.1 MB) in 11.59s (largest was `libz-sys` at 2.5 MB)
  Compiling libc v0.2.139
  :
  Compiling mountpoint-s3 v0.2.0 (/home/abhijit/mountpoint-s3/mountpoint-s3)
  Compiling mountpoint-s3-crt v0.2.0 (/home/abhijit/mountpoint-s3/mountpoint-s3-crt)
  Compiling mountpoint-s3-client v0.2.0 (/home/abhijit/mountpoint-s3/mountpoint-s3-client)
   Finished release [optimized + debuginfo] target(s) in 5m 55s

Release binary will be created at target/release/mount-s3.

abhijit@AwsJunkie:~/mountpoint-s3$ ls -li target/release/mount-s3
112982 -rwxr-xr-x 2 abhijit abhijit 108766920 Mar 15 20:57 target/release/mount-s3

For convenience, let's add the full path of mount-s3 binary to PATH  variable.  Append export PATH=$PATH:/home/abhijit/mountpoint-s3/target/release/mount-s3 at the end of .bashrc. And reload .bashrc.

abhijit@AwsJunkie:~/mountpoint-s3$ sudo nano ~/.bashrc
abhijit@AwsJunkie:~/mountpoint-s3$ source ~/.bashrc

To verify, check the version.

abhijit@AwsJunkie:~/mountpoint-s3$ cd ~
abhijit@AwsJunkie:~$ mount-s3 --version
mountpoint-s3 0.2.0-f15b30c

Configuration

Create ~/.aws/credentials file and add aws_access_key_id and aws_secret_access_key.

abhijit@AwsJunkie:~$ mkdir ~/.aws
abhijit@AwsJunkie:~$ sudo nano ~/.aws/credentials

Sample ~/.aws/credentials content

[default]
aws_access_key_id = AKIAUI2UW222A6I6R6O
aws_secret_access_key = 81926124b-8fcd-4239-9ab0-630f772222

Create a directory (e.g. ~/s3mount) and mount the same with S3 bucket (e.g. awsjunkie-mountpoint-s3).

abhijit@AwsJunkie:~$ mkdir ~/s3mount
abhijit@AwsJunkie:~$ mount-s3 awsjunkie-mountpoint-s3 ~/s3mount

Demo

Try different read-only filesystem commands on the mounted directory. Play around with objects of AWS S3 bucket as if they are there in your local filesystem.

abhijit@AwsJunkie:~$ tree s3mount/
s3mount/
├── a
│   └── x
│       └── y
│           ├── Free AWS Courses and Labs in AWSEducate.mp4
│           ├── Get AWS Certified Professional Challenge 50 percent voucher.mp4
│           └── How to open AWS Educate account.mp4
├── b
│   └── z
│       ├── how-to-install-aws-cli-on-wsl2-windows-subsystem-for-linux-ubuntu-cloud.png
│       └── how-to-install-sam-cli-on-wsl2-ubuntu-linux.png
└── c
    └── Welcome to Mountpoint for Amazon S3.txt

6 directories, 6 files
abhijit@AwsJunkie:~$ ls -l s3mount/a/x/y
total 124807
-rw-r--r-- 1 abhijit abhijit 36084037 Mar 15 22:23 'Free AWS Courses and Labs in AWSEducate.mp4'
-rw-r--r-- 1 abhijit abhijit 65996157 Mar 15 22:23 'Get AWS Certified Professional Challenge 50 percent voucher.mp4'
-rw-r--r-- 1 abhijit abhijit 25721522 Mar 15 22:23 'How to open AWS Educate account.mp4'
abhijit@AwsJunkie:~$ cat s3mount/c/"Welcome to Mountpoint for Amazon S3.txt"
Welcome to Mountpoint for Amazon S3!

For any write operation, we'll see message "Read-only file system".

abhijit@AwsJunkie:~$ touch s3mount/write-test.txt
touch: cannot touch 's3mount/write-test.txt': Read-only file system

Command Usage Information:

abhijit@AwsJunkie:~$ mount-s3 --help
mountpoint-s3 0.2.0-f5521b2
Mountpoint for Amazon S3

USAGE:
    mount-s3 [OPTIONS] <BUCKET_NAME> <MOUNT_POINT>

ARGS:
    <BUCKET_NAME>    Name of bucket to mount
    <MOUNT_POINT>    Mount point for file system

OPTIONS:
        --allow-other
            Allow other non-root users to access file system

        --allow-root
            Allow root user to access file system

        --auto-unmount
            Automatically unmount on exit

        --dir-mode <DIR_MODE>
            Directory permissions [default: 0755]

        --endpoint-url <ENDPOINT_URL>
            Override S3 endpoint URL

    -f, --foreground
            Run as foreground process

        --file-mode <FILE_MODE>
            File permissions [default: 0644]

        --gid <GID>
            Owner GID [default: current user's GID]

    -h, --help
            Print help information

    -l, --log-directory <LOG_DIRECTORY>
            Log file directory. [default: $HOME/.mountpoint-s3]

        --part-size <PART_SIZE>
            Part size for multi-part GET and PUT

        --path-addressing
            Force path-style addressing

        --prefix <PREFIX>
            Prefix inside the bucket to mount. Mounts the entire bucket if unspecified.

        --region <REGION>
            AWS region of the bucket

        --thread-count <N>
            Number of FUSE daemon threads

        --throughput-target-gbps <N (Gbps)>
            Desired throughput in Gbps

        --uid <UID>
            Owner UID [default: current user's UID]

    -V, --version
            Print version information

        --virtual-addressing
            Force virtual-host-style addressing

References

• How to Install Windows Subsystem for Linux (WSL2)
• Mountpoint for Amazon S3 - Github