go/sqlite3
1
0
Fork 0
mirror of https://github.com/ncruces/go-sqlite3.git synced 2026-01-11 21:49:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Nuno Cruces
2024-04-11 13:20:30 +01:00
parent e503be641a
commit 7260962aba
6 changed files with 81 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
README.md
View File

@@ -72,49 +72,10 @@ Go, wazero and [`x/sys`](https://pkg.go.dev/golang.org/x/sys) are the _only_ run
### Caveats
This module replaces the SQLite [OS Interface](https://sqlite.org/vfs.html)
(aka VFS) with a [pure Go](vfs/) implementation.
This has benefits, but also comes with some drawbacks.
(aka VFS) with a [pure Go](vfs/) implementation,
which has advantages and disadvantages.
#### Write-Ahead Logging
Because Wasm does not support shared memory,
[WAL](https://sqlite.org/wal.html) support is [limited](https://sqlite.org/wal.html#noshm).
To work around this limitation, SQLite is [patched](sqlite3/locking_mode.patch)
to always use `EXCLUSIVE` locking mode for WAL databases.
Because connection pooling is incompatible with `EXCLUSIVE` locking mode,
to use the [`database/sql`](https://pkg.go.dev/database/sql) driver
with WAL mode databases you should disable connection pooling by calling
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
#### File Locking
POSIX advisory locks, which SQLite uses on Unix, are
[broken by design](https://sqlite.org/src/artifact/2e8b12?ln=1073-1161).
On Linux, macOS and illumos, this module uses
[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html)
to synchronize access to database files.
OFD locks are fully compatible with POSIX advisory locks.
On BSD Unixes, this module uses
[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2).
On BSD Unixes, BSD locks are fully compatible with POSIX advisory locks.
On Windows, this module uses `LockFileEx` and `UnlockFileEx`,
like SQLite.
On all other platforms, file locking is not supported, and you must use
[`nolock=1`](https://sqlite.org/uri.html#urinolock)
(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable))
to open database files.
You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking)
to check if your platform supports file locking.
To use the [`database/sql`](https://pkg.go.dev/database/sql) driver
with `nolock=1` you must disable connection pooling by calling
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
Read more about the Go VFS design [here](vfs/README.md).
### Testing
@@ -122,7 +83,7 @@ This project aims for [high test coverage](https://github.com/ncruces/go-sqlite3
It also benefits greatly from [SQLite's](https://sqlite.org/testing.html) and
[wazero's](https://tetrate.io/blog/introducing-wazero-from-tetrate/#:~:text=Rock%2Dsolid%20test%20approach) thorough testing.
The pure Go VFS is tested by running SQLite's
The Go VFS is tested by running SQLite's
[mptest](https://github.com/sqlite/sqlite/blob/master/mptest/mptest.c)
on Linux, macOS, Windows and FreeBSD.

2
internal/util/mmap.go
View File

@@ -1,4 +1,4 @@
//go:build (linux || darwin) && (amd64 || arm64) && !sqlite3_flock && !sqlite3_noshm && !sqlite3_nosys
//go:build (darwin || linux || illumos) && (amd64 || arm64) && !sqlite3_flock && !sqlite3_noshm && !sqlite3_nosys
package util

2
internal/util/mmap_other.go
View File

@@ -1,4 +1,4 @@
//go:build !(linux || darwin) || !(amd64 || arm64) || sqlite3_flock || sqlite3_noshm || sqlite3_nosys
//go:build !(darwin || linux || illumos) || !(amd64 || arm64) || sqlite3_flock || sqlite3_noshm || sqlite3_nosys
package util

73
vfs/README.md
View File

@@ -2,6 +2,75 @@
This package implements the SQLite [OS Interface](https://sqlite.org/vfs.html) (aka VFS).
It replaces the default SQLite VFS with a pure Go implementation.
It replaces the default SQLite VFS with a **pure Go** implementation.
It also exposes interfaces that should allow you to implement your own custom VFSes.
It also exposes [interfaces](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#VFS)
that should allow you to implement your own custom VFSes.
Since it is a from scratch reimplementation,
there are naturally some ways it deviates from the original.
The main differences are [file locking](#file-locking) and [WAL mode](write-ahead-logging) support.
### File Locking
POSIX advisory locks, which SQLite uses on Unix, are
[broken by design](https://sqlite.org/src/artifact/2e8b12?ln=1073-1161).
On Linux, macOS and illumos, this module uses
[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html)
to synchronize access to database files.
OFD locks are fully compatible with POSIX advisory locks.
On BSD Unixes, this module uses
[BSD locks](https://man.freebsd.org/cgi/man.cgi?query=flock&sektion=2).
On BSD Unixes, BSD locks are fully compatible with POSIX advisory locks.
However, concurrency is reduced with BSD locks
(`BEGIN IMMEDIATE` behaves the same as `BEGIN EXCLUSIVE`).
On Windows, this module uses `LockFileEx` and `UnlockFileEx`,
like SQLite.
On all other platforms, file locking is not supported, and you must use
[`nolock=1`](https://sqlite.org/uri.html#urinolock)
(or [`immutable=1`](https://sqlite.org/uri.html#uriimmutable))
to open database files.
To use the [`database/sql`](https://pkg.go.dev/database/sql) driver
with `nolock=1` you must disable connection pooling by calling
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
You can use [`vfs.SupportsFileLocking`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsFileLocking)
to check if your platform supports file locking.
### Write-Ahead Logging
On 64-bit Linux, macOS and illumos, this module uses `mmap` to implement
[shared-memory for the WAL-index](https://sqlite.org/wal.html#implementation_of_shared_memory_for_the_wal_index),
like SQLite.
To allow `mmap` to work, each connection needs to reserve a lot of address space.\
To limit the amount of address space each connection needs,
use [`WithMemoryLimitPages`](../tests/parallel/parallel_test.go#L21).
On all other platforms, [WAL](https://sqlite.org/wal.html) support is
[limited](https://sqlite.org/wal.html#noshm).
To work around that limitation, SQLite is [patched](sqlite3/locking_mode.patch)
to automatically use `EXCLUSIVE` locking mode for WAL databases on such platforms.
Because connection pooling is incompatible with `EXCLUSIVE` locking mode,
to use the [`database/sql`](https://pkg.go.dev/database/sql) driver
with WAL mode databases you should disable connection pooling by calling
[`db.SetMaxOpenConns(1)`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns).
You can use [`vfs.SupportsSharedMemory`](https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs#SupportsSharedMemory)
to check if your platform supports shared memory.
### Build tags
The VFS can be customized with a few build tags:
- `sqlite3_flock` forces the use of BSD locks; it can be used on macOS to test the BSD locking implementation.
- `sqlite3_nosys` prevents importing [`x/sys`](https://pkg.go.dev/golang.org/x/sys);
disables locking _and_ shared memory on all platforms.
- `sqlite3_noshm` disables shared memory on all platforms.

3
vfs/os_ofd.go
View File

@@ -3,6 +3,7 @@
package vfs
import (
"math/rand"
"os"
"time"
@@ -43,7 +44,7 @@ func osLock(file *os.File, typ int16, start, len int64, timeout time.Duration, d
if timeout < time.Since(before) {
break
}
osSleep(time.Millisecond)
osSleep(time.Duration(rand.Int63n(int64(time.Millisecond))))
}
}
return osLockErrorCode(err, def)

3
vfs/os_windows.go
View File

@@ -3,6 +3,7 @@
package vfs
import (
"math/rand"
"os"
"time"
@@ -135,7 +136,7 @@ func osLock(file *os.File, flags, start, len uint32, timeout time.Duration, def
if timeout < time.Since(before) {
break
}
osSleep(time.Millisecond)
osSleep(time.Duration(rand.Int63n(int64(time.Millisecond))))
}
}
return osLockErrorCode(err, def)