YouTube video: A New Era for Database Design with TigerBeetle, InfoQ, Sept 21, 2023
Joran Greef, Founder and CEO @TigerBeetle
- New open-source distributed database
- Designed to track the "Movement of Value" e.g. financial transactions - payments, trades
- Mission-Critical Safety and Performance
- Schema support for Balance Tracking via Double-Entry Accounting Primitives out-of-the-box
- Designed for High Availability (HA) with automated failover support
What is Durability?
Once a database transaction has been acknowledge as committed to the user,
it will remain committed, even in the event of a crash.
https://danluu.com/file-consistency
All File Systems Are Not Created Equal:On the Complexity of Crafting Crash-Consistent Applications
At least 3 ways that a database can be designed to write to disk:
-
Direct I/O (O_DIRECT)
- database takes the responsibility for working with the disk
- bypasses the kernel page cache
-
Buffered I/O and fsync()
- outsource durability to kernel with buffered I/O
- this strategy employed by Postgres and several other databases
If fsync() returns an error (EIO), then the database has 3 options:
i. Ignore fsync() error
ii. Retry fsync() until success
iii. Crash and restartThe cracks in Buffered I/O strategy
a. Writing to cache instead of disk => You Lose Congetion Control
b. You Lose the Ability To Prioritize I/O
c. All or nothing => You Lose Fine-Grained Error Handling
d. You Lose Memory Bandwidth• PostgreSQL's fsync() surprise By Jonathan Corbet, April 18, 2018
• PostgreSQL vs. fsync How is it possible that PostgreSQL used fsync incorrectly for 20 years, and what we'll do about it. Tomas Vondra, FOSDEM'19
Q. How Does a Database Provide Crash Consistency Through Power Loss?
A. The Write Ahead Log (WAL) - the crucial building block for ensuring atomic changes
To continue: https://youtu.be/ehYcCTHRyFs?t=1325