diff --git a/README.md b/README.md index 62a1a2e..dc77781 100644 --- a/README.md +++ b/README.md @@ -72,49 +72,10 @@ Go, wazero and [`x/sys`](https://pkg.go.dev/golang.org/x/sys) are the _only_ run ### Caveats This module replaces the SQLite [OS Interface](https://sqlite.org/vfs.html) -(aka VFS) with a [pure Go](vfs/) implementation. -This has benefits, but also comes with some drawbacks. +(aka VFS) with a [pure Go](vfs/) implementation, +which has advantages and disadvantages. -#### Write-Ahead Logging - -Because Wasm does not support shared memory, -[WAL](https://sqlite.org/wal.html) support is [limited](https://sqlite.org/wal.html#noshm). - -To work around this limitation, SQLite is [patched](sqlite3/locking_mode.patch) -to always use `EXCLUSIVE` locking mode for WAL databases. - -Because connection pooling is incompatible with `EXCLUSIVE` locking mode, -to use the [`database/sql`](https://pkg.go.dev/database/sql) driver -with WAL mode databases you should disable connection pooling by calling -[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns). - -#### File Locking - -POSIX advisory locks, which SQLite uses on Unix, are -[broken by design](https://sqlite.org/src/artifact/2e8b12?ln=1073-1161). - -On Linux, macOS and illumos, this module uses -[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html) -to synchronize access to database files. -OFD locks are fully compatible with POSIX advisory locks. - -On BSD Unixes, this module uses -[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2). -On BSD Unixes, BSD locks are fully compatible with POSIX advisory locks. - -On Windows, this module uses `LockFileEx` and `UnlockFileEx`, -like SQLite. - -On all other platforms, file locking is not supported, and you must use -[`nolock=1`](https://sqlite.org/uri.html#urinolock) -(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable)) -to open database files. -You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking) -to check if your platform supports file locking. - -To use the [`database/sql`](https://pkg.go.dev/database/sql) driver -with `nolock=1` you must disable connection pooling by calling -[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns). +Read more about the Go VFS design [here](vfs/README.md). ### Testing @@ -122,7 +83,7 @@ This project aims for [high test coverage](https://github.com/ncruces/go-sqlite3 It also benefits greatly from [SQLite's](https://sqlite.org/testing.html) and [wazero's](https://tetrate.io/blog/introducing-wazero-from-tetrate/#:~:text=Rock%2Dsolid%20test%20approach) thorough testing. -The pure Go VFS is tested by running SQLite's +The Go VFS is tested by running SQLite's [mptest](https://github.com/sqlite/sqlite/blob/master/mptest/mptest.c) on Linux, macOS, Windows and FreeBSD. diff --git a/internal/util/mmap.go b/internal/util/mmap.go index 45ca3f9..c3daaab 100644 --- a/internal/util/mmap.go +++ b/internal/util/mmap.go @@ -1,4 +1,4 @@ -//go:build (linux || darwin) && (amd64 || arm64) && !sqlite3_flock && !sqlite3_noshm && !sqlite3_nosys +//go:build (darwin || linux || illumos) && (amd64 || arm64) && !sqlite3_flock && !sqlite3_noshm && !sqlite3_nosys package util diff --git a/internal/util/mmap_other.go b/internal/util/mmap_other.go index d585340..8e46e83 100644 --- a/internal/util/mmap_other.go +++ b/internal/util/mmap_other.go @@ -1,4 +1,4 @@ -//go:build !(linux || darwin) || !(amd64 || arm64) || sqlite3_flock || sqlite3_noshm || sqlite3_nosys +//go:build !(darwin || linux || illumos) || !(amd64 || arm64) || sqlite3_flock || sqlite3_noshm || sqlite3_nosys package util diff --git a/vfs/README.md b/vfs/README.md index 0d734b4..5d33e59 100644 --- a/vfs/README.md +++ b/vfs/README.md @@ -2,6 +2,75 @@ This package implements the SQLite [OS Interface](https://sqlite.org/vfs.html) (aka VFS). -It replaces the default SQLite VFS with a pure Go implementation. +It replaces the default SQLite VFS with a **pure Go** implementation. -It also exposes interfaces that should allow you to implement your own custom VFSes. \ No newline at end of file +It also exposes [interfaces](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#VFS) +that should allow you to implement your own custom VFSes. + +Since it is a from scratch reimplementation, +there are naturally some ways it deviates from the original. + +The main differences are [file locking](#file-locking) and [WAL mode](write-ahead-logging) support. + +### File Locking + +POSIX advisory locks, which SQLite uses on Unix, are +[broken by design](https://sqlite.org/src/artifact/2e8b12?ln=1073-1161). + +On Linux, macOS and illumos, this module uses +[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html) +to synchronize access to database files. +OFD locks are fully compatible with POSIX advisory locks. + +On BSD Unixes, this module uses +[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2). +On BSD Unixes, BSD locks are fully compatible with POSIX advisory locks. +However, concurrency is reduced with BSD locks +(`BEGIN IMMEDIATE` behaves the same as `BEGIN EXCLUSIVE`). + +On Windows, this module uses `LockFileEx` and `UnlockFileEx`, +like SQLite. + +On all other platforms, file locking is not supported, and you must use +[`nolock=1`](https://sqlite.org/uri.html#urinolock) +(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable)) +to open database files. + +To use the [`database/sql`](https://pkg.go.dev/database/sql) driver +with `nolock=1` you must disable connection pooling by calling +[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns). + +You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking) +to check if your platform supports file locking. + +### Write-Ahead Logging + +On 64-bit Linux, macOS and illumos, this module uses `mmap` to implement +[shared-memory for the WAL-index](https://sqlite.org/wal.html#implementation_of_shared_memory_for_the_wal_index), +like SQLite. + +To allow `mmap` to work, each connection needs to reserve a lot of address space.\ +To limit the amount of address space each connection needs, +use [`WithMemoryLimitPages`](../tests/parallel/parallel_test.go#L21). + +On all other platforms, [WAL](https://sqlite.org/wal.html) support is +[limited](https://sqlite.org/wal.html#noshm). + +To work around that limitation, SQLite is [patched](sqlite3/locking_mode.patch) +to automatically use `EXCLUSIVE` locking mode for WAL databases on such platforms. + +Because connection pooling is incompatible with `EXCLUSIVE` locking mode, +to use the [`database/sql`](https://pkg.go.dev/database/sql) driver +with WAL mode databases you should disable connection pooling by calling +[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns). + +You can use [`vfs.SupportsSharedMemory`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsSharedMemory) +to check if your platform supports shared memory. + +### Build tags + +The VFS can be customized with a few build tags: +- `sqlite3_flock` forces the use of BSD locks; it can be used on macOS to test the BSD locking implementation. +- `sqlite3_nosys` prevents importing [`x/sys`](https://pkg.go.dev/golang.org/x/sys); + disables locking _and_ shared memory on all platforms. +- `sqlite3_noshm` disables shared memory on all platforms. \ No newline at end of file diff --git a/vfs/os_ofd.go b/vfs/os_ofd.go index f6898ed..0fe64d8 100644 --- a/vfs/os_ofd.go +++ b/vfs/os_ofd.go @@ -3,6 +3,7 @@ package vfs import ( + "math/rand" "os" "time" @@ -43,7 +44,7 @@ func osLock(file *os.File, typ int16, start, len int64, timeout time.Duration, d if timeout < time.Since(before) { break } - osSleep(time.Millisecond) + osSleep(time.Duration(rand.Int63n(int64(time.Millisecond)))) } } return osLockErrorCode(err, def) diff --git a/vfs/os_windows.go b/vfs/os_windows.go index 99c91f6..5c68754 100644 --- a/vfs/os_windows.go +++ b/vfs/os_windows.go @@ -3,6 +3,7 @@ package vfs import ( + "math/rand" "os" "time" @@ -135,7 +136,7 @@ func osLock(file *os.File, flags, start, len uint32, timeout time.Duration, def if timeout < time.Since(before) { break } - osSleep(time.Millisecond) + osSleep(time.Duration(rand.Int63n(int64(time.Millisecond)))) } } return osLockErrorCode(err, def)