What is Hyperfile

Hyperfile’s history originates from a very simple and fundamental need: performing file operations directly on S3.

Based on B+tree data structure, Hyperfile implements an object format that enables random read write operations directly on S3. Through file-like interface APIs, applications can perform file operations directly.

Key features:

• Appending data
• Random write
• Truncate
• Write zero aware

Hyperfile project currently focuses on file-level feature support, so it is not a implementation of filesystem. This means that filesystem level operations, such as: mv, mkdir, etc., are currently not supported.

Hyperfile also support following advanced features:

• Customized block size
• Copy-on-Write (CoW)
• Snapshot
• Write-Ahead-Log (WAL)
• local cache for file data and meta data

Typical use cases#

• Virtual disk image
• Dump of virtual memory content (e.g. firecracker)
• Files that are kept open for long periods and have contents keep appending

Components#

• btree-ondisk Implementation of Btree
• hyperfile-reactor A lightweight single-threaded task execution framework
• hyperfile Hyperfile core
• hyperfile-cleaner Hyperfile segments cleanup
• hypercli Command line toolkits
• hypernbd Block device implementation as a NBD device