Compare commits

..

33 Commits

Author SHA1 Message Date
Rod Vagg
fc27a0c8a4 fix: error not as pointer 2023-03-17 11:05:58 +11:00
Henrique Dias
65e3baa1cb feat: wrap parsing errors into ErrInvalidCid 2023-03-16 08:44:03 +01:00
dependabot[bot]
85c423677c build(deps): bump golang.org/x/crypto
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.0.0-20210506145944-38f3c27a63bf to 0.1.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/commits/v0.1.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-27 17:42:17 +11:00
dependabot[bot]
197da0b712 build(deps): bump golang.org/x/sys
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.0.0-20210309074719-68d13333faf2 to 0.1.0.
- [Release notes](https://github.com/golang/sys/releases)
- [Commits](https://github.com/golang/sys/commits/v0.1.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sys
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-27 16:12:56 +11:00
Rod Vagg
06fe289a6a fix: use crypto/rand.Read 2023-02-09 15:30:24 +11:00
web3-bot
a1630a3bd9 update .github/workflows/release-check.yml 2023-02-09 15:30:24 +11:00
web3-bot
225f5db116 update .github/workflows/go-check.yml 2023-02-09 15:30:24 +11:00
web3-bot
c2c040dac7 update .github/workflows/go-test.yml 2023-02-09 15:30:24 +11:00
web3-bot
b729e38c6e bump go.mod to Go 1.19 and run go fix 2023-02-09 15:30:24 +11:00
Nikhilesh Susarla
823c6b8a59 Fix README.md example error (#146) 2022-10-19 17:17:54 +11:00
Jorropo
350d6afb62 chore: release v0.3.2 2022-09-04 16:00:09 +02:00
Jorropo
68090a8273 Revert "fix: bring back, but deprecate CodecToStr and Codecs"
This reverts commit 548365f3a7.
2022-09-04 16:00:09 +02:00
Jorropo
69784e9dd7 chore: release v0.2.1 2022-09-04 13:03:33 +02:00
Rod Vagg
548365f3a7 fix: bring back, but deprecate CodecToStr and Codecs
use go-multicodec as the source of truth
2022-09-04 13:03:33 +02:00
web3-bot
5da28005a9 update .github/workflows/go-check.yml 2022-08-25 15:55:13 +10:00
web3-bot
7f6771bc54 update .github/workflows/go-test.yml 2022-08-25 15:55:13 +10:00
web3-bot
5b6a9dce73 run gofmt -s 2022-08-25 15:55:13 +10:00
web3-bot
890f55d9f3 bump go.mod to Go 1.18 and run go fix 2022-08-25 15:55:13 +10:00
Jorropo
085afa84d0 chore: release v0.3.0 2022-08-25 05:34:36 +02:00
Jorropo
0052a62190 fix: return nil Bytes() if the Cid in undef 2022-08-25 05:34:36 +02:00
GitHub
802b45594e chore: Update .github/workflows/stale.yml [skip ci] 2022-07-21 09:54:55 +00:00
Steven Allen
8f7d7ac18e Merge pull request #139 from mg98/feat/must-parse
Add MustParse
2022-06-27 08:55:37 -07:00
GitHub
b106e0883a Update .github/workflows/stale.yml 2022-06-27 10:45:49 +00:00
Marcel Gregoriadis
386c6cc18a Add MustParse 2022-06-24 17:35:23 +02:00
ipfs-mgmt-read-write[bot]
f4b3e66993 Add .github/workflows/stale.yml 2022-06-13 12:53:24 +00:00
Marcin Rataj
b2064d74a8 fix: remove invalid multicodec2string mappings (#137)
* refactor: remove Codecs table
* chore: go-cid 0.2.0

Codec table was missing dag-json and it had invalid code for dag-cbor.
It also had invalid string representation of dag-pb -- it was using
'protobuf' which is a totally different code.

This removes invalid mappings entirely.
https://github.com/multiformats/go-multicodec should be used instead.
2022-04-21 22:39:31 +02:00
web3-bot
ddd9ef7248 update .github/workflows/go-check.yml 2022-04-05 09:32:38 +10:00
web3-bot
6be8b631d0 update .github/workflows/go-test.yml 2022-04-05 09:32:38 +10:00
web3-bot
395d392889 update .github/workflows/automerge.yml 2022-04-05 09:32:38 +10:00
web3-bot
1d11990417 bump go.mod to Go 1.17 and run go fix 2022-04-05 09:32:38 +10:00
web3-bot
dc3bb41324 sync: update CI config files (#136)
* add version.json file

* update .github/workflows/go-test.yml

* update .github/workflows/go-check.yml

* add .github/workflows/releaser.yml

* add .github/workflows/release-check.yml

* add .github/workflows/tagpush.yml

Co-authored-by: web3-bot <web3-bot@users.noreply.github.com>
2021-12-13 12:26:14 +01:00
Masih H. Derkani
28f4a5eab6 Benchmark existing ways to check for IDENTITY CIDs
Add benchmarks that compare two ways of checking for
`multihash.IDENTITY` code:
1. `Cid.Prefix().MhType`
2. Decode of `Cid.Hash()`

This benchmark illustrates that using Cid.Prefix is more efficient than
`multihash.Decode`. Users wishing to perform such a check should use
`Cid.Prefix`.

Consider that `Cid.Prefix` is already efficient enough and introducing a
dedicated API for performing this check will likely result in small
gains.

Relates to #133
2021-11-08 14:52:45 +00:00
Daniel Martí
5640b0169f avoid double alloc in NewCidV1
We allocate once via "make([]byte, len)",
and again when that buffer is converted to a string.

Thankfully, since Go 1.10 we have strings.Builder,
designed specifically for this use case.

In a downstream benchmark in go-car,
which needs to reconstruct many CID values,
we see small but nice gains:

    name           old time/op    new time/op    delta
    ReadBlocks-16    1.09ms ± 4%    1.06ms ± 5%   -3.33%  (p=0.007 n=11+11)

    name           old speed      new speed      delta
    ReadBlocks-16   478MB/s ± 4%   494MB/s ± 5%   +3.46%  (p=0.007 n=11+11)

    name           old alloc/op   new alloc/op   delta
    ReadBlocks-16    1.30MB ± 0%    1.25MB ± 0%   -3.86%  (p=0.000 n=12+12)

    name           old allocs/op  new allocs/op  delta
    ReadBlocks-16     9.50k ± 0%     8.45k ± 0%  -11.05%  (p=0.000 n=12+12)
2021-09-12 14:08:59 +02:00
19 changed files with 314 additions and 247 deletions

View File

@@ -1,53 +1,11 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.
# Automatically merge pull requests opened by web3-bot, as soon as (and only if) all tests pass.
# This reduces the friction associated with updating with our workflows.
on: [ pull_request ]
name: Automerge
on: [ pull_request ]
jobs:
automerge-check:
if: github.event.pull_request.user.login == 'web3-bot'
runs-on: ubuntu-latest
outputs:
status: ${{ steps.should-automerge.outputs.status }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Check if we should automerge
id: should-automerge
run: |
for commit in $(git rev-list --first-parent origin/${{ github.event.pull_request.base.ref }}..${{ github.event.pull_request.head.sha }}); do
committer=$(git show --format=$'%ce' -s $commit)
echo "Committer: $committer"
if [[ "$committer" != "web3-bot@users.noreply.github.com" ]]; then
echo "Commit $commit wasn't committed by web3-bot, but by $committer."
echo "::set-output name=status::false"
exit
fi
done
echo "::set-output name=status::true"
automerge:
needs: automerge-check
runs-on: ubuntu-latest
# The check for the user is redundant here, as this job depends on the automerge-check job,
# but it prevents this job from spinning up, just to be skipped shortly after.
if: github.event.pull_request.user.login == 'web3-bot' && needs.automerge-check.outputs.status == 'true'
steps:
- name: Wait on tests
uses: lewagon/wait-on-check-action@bafe56a6863672c681c3cf671f5e10b20abf2eaa # v0.2
with:
ref: ${{ github.event.pull_request.head.sha }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
wait-interval: 10
running-workflow-name: 'automerge' # the name of this job
- name: Merge PR
uses: pascalgn/automerge-action@741c311a47881be9625932b0a0de1b0937aab1ae # v0.13.1
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
MERGE_LABELS: ""
MERGE_METHOD: "squash"
MERGE_DELETE_BRANCH: true
uses: protocol/.github/.github/workflows/automerge.yml@master
with:
job: 'automerge'

View File

@@ -8,26 +8,20 @@ jobs:
unit:
runs-on: ubuntu-latest
name: All
env:
RUNGOGENERATE: false
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
submodules: recursive
- uses: actions/setup-go@v2
- id: config
uses: protocol/.github/.github/actions/read-config@master
- uses: actions/setup-go@v3
with:
go-version: "1.17.x"
go-version: 1.20.x
- name: Run repo-specific setup
uses: ./.github/actions/go-check-setup
if: hashFiles('./.github/actions/go-check-setup') != ''
- name: Read config
if: hashFiles('./.github/workflows/go-check-config.json') != ''
run: |
if jq -re .gogenerate ./.github/workflows/go-check-config.json; then
echo "RUNGOGENERATE=true" >> $GITHUB_ENV
fi
- name: Install staticcheck
run: go install honnef.co/go/tools/cmd/staticcheck@df71e5d0e0ed317ebf43e6e59cf919430fa4b8f2 # 2021.1.1 (v0.2.1)
run: go install honnef.co/go/tools/cmd/staticcheck@4970552d932f48b71485287748246cf3237cebdf # 2023.1 (v0.4.0)
- name: Check that go.mod is tidy
uses: protocol/multiple-go-modules@v1.2
with:
@@ -39,7 +33,7 @@ jobs:
fi
git diff --exit-code -- go.sum go.mod
- name: gofmt
if: ${{ success() || failure() }} # run this step even if the previous one failed
if: success() || failure() # run this step even if the previous one failed
run: |
out=$(gofmt -s -l .)
if [[ -n "$out" ]]; then
@@ -47,12 +41,12 @@ jobs:
exit 1
fi
- name: go vet
if: ${{ success() || failure() }} # run this step even if the previous one failed
if: success() || failure() # run this step even if the previous one failed
uses: protocol/multiple-go-modules@v1.2
with:
run: go vet ./...
- name: staticcheck
if: ${{ success() || failure() }} # run this step even if the previous one failed
if: success() || failure() # run this step even if the previous one failed
uses: protocol/multiple-go-modules@v1.2
with:
run: |
@@ -60,15 +54,14 @@ jobs:
staticcheck ./... | sed -e 's@\(.*\)\.go@./\1.go@g'
- name: go generate
uses: protocol/multiple-go-modules@v1.2
if: (success() || failure()) && env.RUNGOGENERATE == 'true'
if: (success() || failure()) && fromJSON(steps.config.outputs.json).gogenerate == true
with:
run: |
git clean -fd # make sure there aren't untracked files / directories
go generate ./...
go generate -x ./...
# check if go generate modified or added any files
if ! $(git add . && git diff-index HEAD --exit-code --quiet); then
echo "go generated caused changes to the repository:"
git status --short
exit 1
fi

View File

@@ -10,38 +10,59 @@ jobs:
fail-fast: false
matrix:
os: [ "ubuntu", "windows", "macos" ]
go: [ "1.16.x", "1.17.x" ]
go: ["1.19.x","1.20.x"]
env:
COVERAGES: ""
runs-on: ${{ matrix.os }}-latest
name: ${{ matrix.os}} (go ${{ matrix.go }})
runs-on: ${{ fromJSON(vars[format('UCI_GO_TEST_RUNNER_{0}', matrix.os)] || format('"{0}-latest"', matrix.os)) }}
name: ${{ matrix.os }} (go ${{ matrix.go }})
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
submodules: recursive
- uses: actions/setup-go@v2
- id: config
uses: protocol/.github/.github/actions/read-config@master
- uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- name: Go information
run: |
go version
go env
- name: Use msys2 on windows
if: matrix.os == 'windows'
shell: bash
# The executable for msys2 is also called bash.cmd
# https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#shells
# If we prepend its location to the PATH
# subsequent 'shell: bash' steps will use msys2 instead of gitbash
run: echo "C:/msys64/usr/bin" >> $GITHUB_PATH
- name: Run repo-specific setup
uses: ./.github/actions/go-test-setup
if: hashFiles('./.github/actions/go-test-setup') != ''
- name: Run tests
if: contains(fromJSON(steps.config.outputs.json).skipOSes, matrix.os) == false
uses: protocol/multiple-go-modules@v1.2
with:
run: go test -v -coverprofile module-coverage.txt ./...
# Use -coverpkg=./..., so that we include cross-package coverage.
# If package ./A imports ./B, and ./A's tests also cover ./B,
# this means ./B's coverage will be significantly higher than 0%.
run: go test -v -shuffle=on -coverprofile=module-coverage.txt -coverpkg=./... ./...
- name: Run tests (32 bit)
if: ${{ matrix.os != 'macos' }} # can't run 32 bit tests on OSX.
# can't run 32 bit tests on OSX.
if: matrix.os != 'macos' &&
fromJSON(steps.config.outputs.json).skip32bit != true &&
contains(fromJSON(steps.config.outputs.json).skipOSes, matrix.os) == false
uses: protocol/multiple-go-modules@v1.2
env:
GOARCH: 386
with:
run: go test -v ./...
run: |
export "PATH=$PATH_386:$PATH"
go test -v -shuffle=on ./...
- name: Run tests with race detector
if: ${{ matrix.os == 'ubuntu' }} # speed things up. Windows and OSX VMs are slow
# speed things up. Windows and OSX VMs are slow
if: matrix.os == 'ubuntu' &&
contains(fromJSON(steps.config.outputs.json).skipOSes, matrix.os) == false
uses: protocol/multiple-go-modules@v1.2
with:
run: go test -v -race ./...
@@ -49,7 +70,7 @@ jobs:
shell: bash
run: echo "COVERAGES=$(find . -type f -name 'module-coverage.txt' | tr -s '\n' ',' | sed 's/,$//')" >> $GITHUB_ENV
- name: Upload coverage to Codecov
uses: codecov/codecov-action@51d810878be5422784e86451c0e7c14e5860ec47 # v2.0.2
uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70 # v3.1.1
with:
files: '${{ env.COVERAGES }}'
env_vars: OS=${{ matrix.os }}, GO=${{ matrix.go }}

13
.github/workflows/release-check.yml vendored Normal file
View File

@@ -0,0 +1,13 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.
name: Release Checker
on:
pull_request_target:
paths: [ 'version.json' ]
jobs:
release-check:
uses: protocol/.github/.github/workflows/release-check.yml@master
with:
go-version: 1.20.x

11
.github/workflows/releaser.yml vendored Normal file
View File

@@ -0,0 +1,11 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.
name: Releaser
on:
push:
paths: [ 'version.json' ]
jobs:
releaser:
uses: protocol/.github/.github/workflows/releaser.yml@master

26
.github/workflows/stale.yml vendored Normal file
View File

@@ -0,0 +1,26 @@
name: Close and mark stale issue
on:
schedule:
- cron: '0 0 * * *'
jobs:
stale:
runs-on: ubuntu-latest
permissions:
issues: write
pull-requests: write
steps:
- uses: actions/stale@v3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.'
close-issue-message: 'This issue was closed because it is missing author input.'
stale-issue-label: 'kind/stale'
any-of-labels: 'need/author-input'
exempt-issue-labels: 'need/triage,need/community-input,need/maintainer-input,need/maintainers-input,need/analysis,status/blocked,status/in-progress,status/ready,status/deferred,status/inactive'
days-before-issue-stale: 6
days-before-issue-close: 7
enable-statistics: true

12
.github/workflows/tagpush.yml vendored Normal file
View File

@@ -0,0 +1,12 @@
# File managed by web3-bot. DO NOT EDIT.
# See https://github.com/protocol/.github/ for details.
name: Tag Push Checker
on:
push:
tags:
- v*
jobs:
releaser:
uses: protocol/.github/.github/workflows/tagpush.yml@master

View File

@@ -59,10 +59,17 @@ fmt.Println("Got CID: ", c)
#### Creating a CID from scratch
```go
import (
cid "github.com/ipfs/go-cid"
mc "github.com/multiformats/go-multicodec"
mh "github.com/multiformats/go-multihash"
)
// Create a cid manually by specifying the 'prefix' parameters
pref := cid.Prefix{
Version: 1,
Codec: cid.Raw,
Codec: uint64(mc.Raw),
MhType: mh.SHA2_256,
MhLength: -1, // default length
}

View File

@@ -39,7 +39,8 @@ type Cid interface {
// and the Multihash length. It does not contains
// any actual content information.
// NOTE: The use -1 in MhLength to mean default length is deprecated,
// use the V0Builder or V1Builder structures instead
//
// use the V0Builder or V1Builder structures instead
type Prefix struct {
Version uint64
Codec uint64

View File

@@ -12,14 +12,14 @@ import (
//
// Sample results on linux amd64 go1.11beta:
//
// BenchmarkCidMap_CidStr-8 100000 16317 ns/op
// BenchmarkCidMap_CidIface-8 100000 20516 ns/op
// BenchmarkCidMap_CidStr-8 100000 16317 ns/op
// BenchmarkCidMap_CidIface-8 100000 20516 ns/op
//
// With benchmem on:
//
// BenchmarkCidMap_CidStr-8 100000 15579 ns/op 11223 B/op 207 allocs/op
// BenchmarkCidMap_CidIface-8 100000 19500 ns/op 12824 B/op 307 allocs/op
// BenchmarkCidMap_StrPlusHax-8 200000 10451 ns/op 7589 B/op 202 allocs/op
// BenchmarkCidMap_CidStr-8 100000 15579 ns/op 11223 B/op 207 allocs/op
// BenchmarkCidMap_CidIface-8 100000 19500 ns/op 12824 B/op 307 allocs/op
// BenchmarkCidMap_StrPlusHax-8 200000 10451 ns/op 7589 B/op 202 allocs/op
//
// We can see here that the impact of interface boxing is significant:
// it increases the time taken to do the inserts to 133%, largely because
@@ -36,7 +36,6 @@ import (
// re-arranges itself, it involves more or less an O(n) copy of the content
// in addition to the alloc itself). This isn't topical to the question of
// whether or not interfaces are a good idea; just for contextualizing.
//
func BenchmarkCidMap_CidStr(b *testing.B) {
for i := 0; i < b.N; i++ {
mp := map[CidStr]int{}

View File

@@ -114,7 +114,7 @@ func NewCidStr(version uint64, codecType uint64, mhash mh.Multihash) CidStr {
//
// For CidV1, the data buffer is in the form:
//
// <version><codec-type><multihash>
// <version><codec-type><multihash>
//
// CidV0 are also supported. In particular, data buffers starting
// with length 34 bytes, which starts with bytes [18,32...] are considered

View File

@@ -110,7 +110,7 @@ func (c CidStruct) Prefix() Prefix {
//
// For CidV1, the data buffer is in the form:
//
// <version><codec-type><multihash>
// <version><codec-type><multihash>
//
// CidV0 are also supported. In particular, data buffers starting
// with length 34 bytes, which starts with bytes [18,32...] are considered

61
benchmark_test.go Normal file
View File

@@ -0,0 +1,61 @@
package cid_test
import (
"math/rand"
"testing"
"github.com/ipfs/go-cid"
"github.com/multiformats/go-multihash"
)
// BenchmarkIdentityCheck benchmarks two ways of checking whether a CIDv1 has multihash.IDENTITY
// code:
// 1. Cid.Prefix(), and
// 2. decoding the Cid.Hash().
//
// This benchmark illustrates that using Cid.Prefix is more efficient than multihash.Decode.
// Users wishing to perform such a check should use Cid.Prefix.
//
// Consider that `Cid.Prefix` is already efficient enough and introducing a dedicated API for
// performing this check will likely result in small gains.
func BenchmarkIdentityCheck(b *testing.B) {
rng := rand.New(rand.NewSource(1413))
data := make([]byte, rng.Intn(100)+1024)
if _, err := rng.Read(data); err != nil {
b.Fatal(err)
}
mh, err := multihash.Sum(data, multihash.IDENTITY, -1)
if err != nil {
b.Fatal(err)
}
cv1 := cid.NewCidV1(cid.Raw, mh)
b.SetBytes(int64(cv1.ByteLen()))
b.ReportAllocs()
b.ResetTimer()
b.Run("Prefix", func(b *testing.B) {
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
if cv1.Prefix().MhType != multihash.IDENTITY {
b.Fatal("expected IDENTITY CID")
}
}
})
})
b.Run("MultihashDecode", func(b *testing.B) {
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
dmh, err := multihash.Decode(cv1.Hash())
if err != nil {
b.Fatal(err)
}
if dmh.Code != multihash.IDENTITY {
b.Fatal("expected IDENTITY CID")
}
}
})
})
}

182
cid.go
View File

@@ -10,7 +10,7 @@
//
// A CIDv1 has four parts:
//
// <cidv1> ::= <multibase-prefix><cid-version><multicodec-packed-content-type><multihash-content-address>
// <cidv1> ::= <multibase-prefix><cid-version><multicodec-packed-content-type><multihash-content-address>
//
// As shown above, the CID implementation relies heavily on Multiformats,
// particularly Multibase
@@ -37,29 +37,53 @@ import (
// UnsupportedVersionString just holds an error message
const UnsupportedVersionString = "<unsupported cid version>"
// ErrInvalidCid is an error that indicates that a CID is invalid.
type ErrInvalidCid struct {
Err error
}
func (e ErrInvalidCid) Error() string {
return fmt.Sprintf("invalid cid: %s", e.Err)
}
func (e ErrInvalidCid) Unwrap() error {
return e.Err
}
func (e ErrInvalidCid) Is(err error) bool {
switch err.(type) {
case ErrInvalidCid, *ErrInvalidCid:
return true
default:
return false
}
}
var (
// ErrCidTooShort means that the cid passed to decode was not long
// enough to be a valid Cid
ErrCidTooShort = errors.New("cid too short")
ErrCidTooShort = ErrInvalidCid{errors.New("cid too short")}
// ErrInvalidEncoding means that selected encoding is not supported
// by this Cid version
ErrInvalidEncoding = errors.New("invalid base encoding")
)
// These are multicodec-packed content types. The should match
// the codes described in the authoritative document:
// https://github.com/multiformats/multicodec/blob/master/table.csv
// Consts below are DEPRECATED and left only for legacy reasons:
// <https://github.com/ipfs/go-cid/pull/137>
// Modern code should use consts from go-multicodec instead:
// <https://github.com/multiformats/go-multicodec>
const (
Raw = 0x55
// common ones
Raw = 0x55
DagProtobuf = 0x70 // https://ipld.io/docs/codecs/known/dag-pb/
DagCBOR = 0x71 // https://ipld.io/docs/codecs/known/dag-cbor/
DagJSON = 0x0129 // https://ipld.io/docs/codecs/known/dag-json/
Libp2pKey = 0x72 // https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#peer-ids
DagProtobuf = 0x70
DagCBOR = 0x71
Libp2pKey = 0x72
GitRaw = 0x78
DagJOSE = 0x85
// other
GitRaw = 0x78
DagJOSE = 0x85 // https://ipld.io/specs/codecs/dag-jose/spec/
EthBlock = 0x90
EthBlockList = 0x91
EthTxTrie = 0x92
@@ -81,64 +105,6 @@ const (
FilCommitmentSealed = 0xf102
)
// Codecs maps the name of a codec to its type
var Codecs = map[string]uint64{
"v0": DagProtobuf,
"raw": Raw,
"protobuf": DagProtobuf,
"cbor": DagCBOR,
"libp2p-key": Libp2pKey,
"git-raw": GitRaw,
"eth-block": EthBlock,
"eth-block-list": EthBlockList,
"eth-tx-trie": EthTxTrie,
"eth-tx": EthTx,
"eth-tx-receipt-trie": EthTxReceiptTrie,
"eth-tx-receipt": EthTxReceipt,
"eth-state-trie": EthStateTrie,
"eth-account-snapshot": EthAccountSnapshot,
"eth-storage-trie": EthStorageTrie,
"bitcoin-block": BitcoinBlock,
"bitcoin-tx": BitcoinTx,
"zcash-block": ZcashBlock,
"zcash-tx": ZcashTx,
"decred-block": DecredBlock,
"decred-tx": DecredTx,
"dash-block": DashBlock,
"dash-tx": DashTx,
"fil-commitment-unsealed": FilCommitmentUnsealed,
"fil-commitment-sealed": FilCommitmentSealed,
"dag-jose": DagJOSE,
}
// CodecToStr maps the numeric codec to its name
var CodecToStr = map[uint64]string{
Raw: "raw",
DagProtobuf: "protobuf",
DagCBOR: "cbor",
GitRaw: "git-raw",
EthBlock: "eth-block",
EthBlockList: "eth-block-list",
EthTxTrie: "eth-tx-trie",
EthTx: "eth-tx",
EthTxReceiptTrie: "eth-tx-receipt-trie",
EthTxReceipt: "eth-tx-receipt",
EthStateTrie: "eth-state-trie",
EthAccountSnapshot: "eth-account-snapshot",
EthStorageTrie: "eth-storage-trie",
BitcoinBlock: "bitcoin-block",
BitcoinTx: "bitcoin-tx",
ZcashBlock: "zcash-block",
ZcashTx: "zcash-tx",
DecredBlock: "decred-block",
DecredTx: "decred-tx",
DashBlock: "dash-block",
DashTx: "dash-tx",
FilCommitmentUnsealed: "fil-commitment-unsealed",
FilCommitmentSealed: "fil-commitment-sealed",
DagJOSE: "dag-jose",
}
// tryNewCidV0 tries to convert a multihash into a CIDv0 CID and returns an
// error on failure.
func tryNewCidV0(mhash mh.Multihash) (Cid, error) {
@@ -146,10 +112,10 @@ func tryNewCidV0(mhash mh.Multihash) (Cid, error) {
// incorrectly detect it as CidV1 in the Version() method
dec, err := mh.Decode(mhash)
if err != nil {
return Undef, err
return Undef, ErrInvalidCid{err}
}
if dec.Code != mh.SHA2_256 || dec.Length != 32 {
return Undef, fmt.Errorf("invalid hash for cidv0 %d-%d", dec.Code, dec.Length)
return Undef, ErrInvalidCid{fmt.Errorf("invalid hash for cidv0 %d-%d", dec.Code, dec.Length)}
}
return Cid{string(mhash)}, nil
}
@@ -233,14 +199,23 @@ func Parse(v interface{}) (Cid, error) {
case Cid:
return v2, nil
default:
return Undef, fmt.Errorf("can't parse %+v as Cid", v2)
return Undef, ErrInvalidCid{fmt.Errorf("can't parse %+v as Cid", v2)}
}
}
// MustParse calls Parse but will panic on error.
func MustParse(v interface{}) Cid {
c, err := Parse(v)
if err != nil {
panic(err)
}
return c
}
// Decode parses a Cid-encoded string and returns a Cid object.
// For CidV1, a Cid-encoded string is primarily a multibase string:
//
// <multibase-type-code><base-encoded-string>
// <multibase-type-code><base-encoded-string>
//
// The base-encoded string represents a:
//
@@ -257,7 +232,7 @@ func Decode(v string) (Cid, error) {
if len(v) == 46 && v[:2] == "Qm" {
hash, err := mh.FromB58String(v)
if err != nil {
return Undef, err
return Undef, ErrInvalidCid{err}
}
return tryNewCidV0(hash)
@@ -265,7 +240,7 @@ func Decode(v string) (Cid, error) {
_, data, err := mbase.Decode(v)
if err != nil {
return Undef, err
return Undef, ErrInvalidCid{err}
}
return Cast(data)
@@ -287,7 +262,7 @@ func ExtractEncoding(v string) (mbase.Encoding, error) {
// check encoding is valid
_, err := mbase.NewEncoder(encoding)
if err != nil {
return -1, err
return -1, ErrInvalidCid{err}
}
return encoding, nil
@@ -296,7 +271,7 @@ func ExtractEncoding(v string) (mbase.Encoding, error) {
// Cast takes a Cid data slice, parses it and returns a Cid.
// For CidV1, the data buffer is in the form:
//
// <version><codec-type><multihash>
// <version><codec-type><multihash>
//
// CidV0 are also supported. In particular, data buffers starting
// with length 34 bytes, which starts with bytes [18,32...] are considered
@@ -307,11 +282,11 @@ func ExtractEncoding(v string) (mbase.Encoding, error) {
func Cast(data []byte) (Cid, error) {
nr, c, err := CidFromBytes(data)
if err != nil {
return Undef, err
return Undef, ErrInvalidCid{err}
}
if nr != len(data) {
return Undef, fmt.Errorf("trailing bytes in data buffer passed to cid Cast")
return Undef, ErrInvalidCid{fmt.Errorf("trailing bytes in data buffer passed to cid Cast")}
}
return c, nil
@@ -425,7 +400,13 @@ func (c Cid) Hash() mh.Multihash {
// Bytes returns the byte representation of a Cid.
// The output of bytes can be parsed back into a Cid
// with Cast().
//
// If c.Defined() == false, it return a nil slice and may not
// be parsable with Cast().
func (c Cid) Bytes() []byte {
if !c.Defined() {
return nil
}
return []byte(c.str)
}
@@ -506,7 +487,7 @@ func (c *Cid) UnmarshalJSON(b []byte) error {
// MarshalJSON procudes a JSON representation of a Cid, which looks as follows:
//
// { "/": "<cid-string>" }
// { "/": "<cid-string>" }
//
// Note that this formatting comes from the IPLD specification
// (https://github.com/ipld/specs/tree/master/ipld)
@@ -563,7 +544,8 @@ func (c Cid) Prefix() Prefix {
// and the Multihash length. It does not contains
// any actual content information.
// NOTE: The use -1 in MhLength to mean default length is deprecated,
// use the V0Builder or V1Builder structures instead
//
// use the V0Builder or V1Builder structures instead
type Prefix struct {
Version uint64
Codec uint64
@@ -602,7 +584,7 @@ func (p Prefix) Sum(data []byte) (Cid, error) {
// Bytes returns a byte representation of a Prefix. It looks like:
//
// <version><codec><mh-type><mh-length>
// <version><codec><mh-type><mh-length>
func (p Prefix) Bytes() []byte {
size := varint.UvarintSize(p.Version)
size += varint.UvarintSize(p.Codec)
@@ -655,12 +637,12 @@ func PrefixFromBytes(buf []byte) (Prefix, error) {
func CidFromBytes(data []byte) (int, Cid, error) {
if len(data) > 2 && data[0] == mh.SHA2_256 && data[1] == 32 {
if len(data) < 34 {
return 0, Undef, fmt.Errorf("not enough bytes for cid v0")
return 0, Undef, ErrInvalidCid{fmt.Errorf("not enough bytes for cid v0")}
}
h, err := mh.Cast(data[:34])
if err != nil {
return 0, Undef, err
return 0, Undef, ErrInvalidCid{err}
}
return 34, Cid{string(h)}, nil
@@ -668,21 +650,21 @@ func CidFromBytes(data []byte) (int, Cid, error) {
vers, n, err := varint.FromUvarint(data)
if err != nil {
return 0, Undef, err
return 0, Undef, ErrInvalidCid{err}
}
if vers != 1 {
return 0, Undef, fmt.Errorf("expected 1 as the cid version number, got: %d", vers)
return 0, Undef, ErrInvalidCid{fmt.Errorf("expected 1 as the cid version number, got: %d", vers)}
}
_, cn, err := varint.FromUvarint(data[n:])
if err != nil {
return 0, Undef, err
return 0, Undef, ErrInvalidCid{err}
}
mhnr, _, err := mh.MHFromBytes(data[n+cn:])
if err != nil {
return 0, Undef, err
return 0, Undef, ErrInvalidCid{err}
}
l := n + cn + mhnr
@@ -745,32 +727,32 @@ func CidFromReader(r io.Reader) (int, Cid, error) {
// The varint package wants a io.ByteReader, so we must wrap our io.Reader.
vers, err := varint.ReadUvarint(br)
if err != nil {
return len(br.dst), Undef, err
return len(br.dst), Undef, ErrInvalidCid{err}
}
// If we have a CIDv0, read the rest of the bytes and cast the buffer.
if vers == mh.SHA2_256 {
if n, err := io.ReadFull(r, br.dst[1:34]); err != nil {
return len(br.dst) + n, Undef, err
return len(br.dst) + n, Undef, ErrInvalidCid{err}
}
br.dst = br.dst[:34]
h, err := mh.Cast(br.dst)
if err != nil {
return len(br.dst), Undef, err
return len(br.dst), Undef, ErrInvalidCid{err}
}
return len(br.dst), Cid{string(h)}, nil
}
if vers != 1 {
return len(br.dst), Undef, fmt.Errorf("expected 1 as the cid version number, got: %d", vers)
return len(br.dst), Undef, ErrInvalidCid{fmt.Errorf("expected 1 as the cid version number, got: %d", vers)}
}
// CID block encoding multicodec.
_, err = varint.ReadUvarint(br)
if err != nil {
return len(br.dst), Undef, err
return len(br.dst), Undef, ErrInvalidCid{err}
}
// We could replace most of the code below with go-multihash's ReadMultihash.
@@ -781,19 +763,19 @@ func CidFromReader(r io.Reader) (int, Cid, error) {
// Multihash hash function code.
_, err = varint.ReadUvarint(br)
if err != nil {
return len(br.dst), Undef, err
return len(br.dst), Undef, ErrInvalidCid{err}
}
// Multihash digest length.
mhl, err := varint.ReadUvarint(br)
if err != nil {
return len(br.dst), Undef, err
return len(br.dst), Undef, ErrInvalidCid{err}
}
// Refuse to make large allocations to prevent OOMs due to bugs.
const maxDigestAlloc = 32 << 20 // 32MiB
if mhl > maxDigestAlloc {
return len(br.dst), Undef, fmt.Errorf("refusing to allocate %d bytes for a digest", mhl)
return len(br.dst), Undef, ErrInvalidCid{fmt.Errorf("refusing to allocate %d bytes for a digest", mhl)}
}
// Fine to convert mhl to int, given maxDigestAlloc.
@@ -812,7 +794,7 @@ func CidFromReader(r io.Reader) (int, Cid, error) {
if n, err := io.ReadFull(r, br.dst[prefixLength:cidLength]); err != nil {
// We can't use len(br.dst) here,
// as we've only read n bytes past prefixLength.
return prefixLength + n, Undef, err
return prefixLength + n, Undef, ErrInvalidCid{err}
}
// This simply ensures the multihash is valid.
@@ -820,7 +802,7 @@ func CidFromReader(r io.Reader) (int, Cid, error) {
// for now, it helps ensure consistency with CidFromBytes.
_, _, err = mh.MHFromBytes(br.dst[mhStart:])
if err != nil {
return len(br.dst), Undef, err
return len(br.dst), Undef, ErrInvalidCid{err}
}
return len(br.dst), Cid{string(br.dst)}, nil

View File

@@ -1,5 +1,4 @@
//go:build gofuzz
// +build gofuzz
package cid

View File

@@ -2,7 +2,9 @@ package cid
import (
"bytes"
crand "crypto/rand"
"encoding/json"
"errors"
"fmt"
"io"
"math/rand"
@@ -15,37 +17,6 @@ import (
mh "github.com/multiformats/go-multihash"
)
// Copying the "silly test" idea from
// https://github.com/multiformats/go-multihash/blob/7aa9f26a231c6f34f4e9fad52bf580fd36627285/multihash_test.go#L13
// Makes it so changing the table accidentally has to happen twice.
var tCodecs = map[uint64]string{
Raw: "raw",
DagProtobuf: "protobuf",
DagCBOR: "cbor",
Libp2pKey: "libp2p-key",
GitRaw: "git-raw",
EthBlock: "eth-block",
EthBlockList: "eth-block-list",
EthTxTrie: "eth-tx-trie",
EthTx: "eth-tx",
EthTxReceiptTrie: "eth-tx-receipt-trie",
EthTxReceipt: "eth-tx-receipt",
EthStateTrie: "eth-state-trie",
EthAccountSnapshot: "eth-account-snapshot",
EthStorageTrie: "eth-storage-trie",
BitcoinBlock: "bitcoin-block",
BitcoinTx: "bitcoin-tx",
ZcashBlock: "zcash-block",
ZcashTx: "zcash-tx",
DecredBlock: "decred-block",
DecredTx: "decred-tx",
DashBlock: "dash-block",
DashTx: "dash-tx",
FilCommitmentUnsealed: "fil-commitment-unsealed",
FilCommitmentSealed: "fil-commitment-sealed",
DagJOSE: "dag-jose",
}
func assertEqual(t *testing.T, a, b Cid) {
if a.Type() != b.Type() {
t.Fatal("mismatch on type")
@@ -60,26 +31,6 @@ func assertEqual(t *testing.T, a, b Cid) {
}
}
func TestTable(t *testing.T) {
if len(tCodecs) != len(Codecs)-1 {
t.Errorf("Item count mismatch in the Table of Codec. Should be %d, got %d", len(tCodecs)+1, len(Codecs))
}
for k, v := range tCodecs {
if Codecs[v] != k {
t.Errorf("Table mismatch: 0x%x %s", k, v)
}
}
}
// The table returns cid.DagProtobuf for "v0"
// so we test it apart
func TestTableForV0(t *testing.T) {
if Codecs["v0"] != DagProtobuf {
t.Error("Table mismatch: Codecs[\"v0\"] should resolve to DagProtobuf (0x70)")
}
}
func TestPrefixSum(t *testing.T) {
// Test creating CIDs both manually and with Prefix.
// Tests: https://github.com/ipfs/go-cid/issues/83
@@ -277,6 +228,9 @@ func TestEmptyString(t *testing.T) {
if err == nil {
t.Fatal("shouldnt be able to parse an empty cid")
}
if !errors.Is(err, ErrInvalidCid{}) {
t.Fatal("error must be ErrInvalidCid")
}
}
func TestV0Handling(t *testing.T) {
@@ -332,6 +286,9 @@ func TestV0ErrorCases(t *testing.T) {
if err == nil {
t.Fatal("should have failed to decode that ref")
}
if !errors.Is(err, ErrInvalidCid{}) {
t.Fatal("error must be ErrInvalidCid")
}
}
func TestNewPrefixV1(t *testing.T) {
@@ -492,7 +449,7 @@ func TestFuzzCid(t *testing.T) {
buf := make([]byte, 128)
for i := 0; i < 200; i++ {
s := rand.Intn(128)
rand.Read(buf[:s])
crand.Read(buf[:s])
_, _ = Cast(buf[:s])
}
}
@@ -505,6 +462,9 @@ func TestParse(t *testing.T) {
if !strings.Contains(err.Error(), "can't parse 123 as Cid") {
t.Fatalf("expected int error, got %s", err.Error())
}
if !errors.Is(err, ErrInvalidCid{}) {
t.Fatalf("expected ErrInvalidCid, got %s", err.Error())
}
theHash := "QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n"
h, err := mh.FromB58String(theHash)
@@ -799,6 +759,9 @@ func TestBadParse(t *testing.T) {
if err == nil {
t.Fatal("expected to fail to parse an invalid CIDv1 CID")
}
if !errors.Is(err, ErrInvalidCid{}) {
t.Fatal("error must be ErrInvalidCid")
}
}
func TestLoggable(t *testing.T) {
@@ -813,3 +776,15 @@ func TestLoggable(t *testing.T) {
t.Fatalf("did not get expected loggable form (got %v)", actual)
}
}
func TestErrInvalidCid(t *testing.T) {
_, err := Decode("not-a-cid")
if err == nil {
t.Fatal("expected error")
}
is := errors.Is(err, ErrInvalidCid{})
if !is {
t.Fatal("expected error to be ErrInvalidCid")
}
}

14
go.mod
View File

@@ -4,7 +4,17 @@ require (
github.com/multiformats/go-multibase v0.0.3
github.com/multiformats/go-multihash v0.0.15
github.com/multiformats/go-varint v0.0.6
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf // indirect
)
go 1.16
require (
github.com/klauspost/cpuid/v2 v2.0.4 // indirect
github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-base32 v0.0.3 // indirect
github.com/multiformats/go-base36 v0.1.0 // indirect
golang.org/x/crypto v0.1.0 // indirect
golang.org/x/sys v0.1.0 // indirect
)
go 1.19

12
go.sum
View File

@@ -19,17 +19,13 @@ github.com/multiformats/go-varint v0.0.6 h1:gk85QWKxh3TazbLxED/NlDVv8+q+ReFJk7Y2
github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf h1:B2n+Zi5QeYRDAEodEu72OS36gmTWjgpXr2+cWcBW90o=
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
golang.org/x/crypto v0.1.0 h1:MDRAIl0xIo9Io2xV565hzXHw3zVseKrJKodhohM5CjU=
golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210309074719-68d13333faf2 h1:46ULzRKLh1CwgRq2dC5SlBzEqqNCi8rreOZnNrbqcIY=
golang.org/x/sys v0.0.0-20210309074719-68d13333faf2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=

3
version.json Normal file
View File

@@ -0,0 +1,3 @@
{
"version": "v0.3.2"
}