fix: remove invalid multicodec2string mappings (#137 )

* refactor: remove Codecs table * chore: go-cid 0.2.0 Codec table was missing dag-json and it had invalid code for dag-cbor. It also had invalid string representation of dag-pb -- it was using 'protobuf' which is a totally different code. This removes invalid mappings entirely. https://github.com/multiformats/go-multicodec should be used instead.
update .github/workflows/go-check.yml
2022-04-21 22:39:31 +02:00 · 2022-04-05 09:32:38 +10:00 · 2022-04-05 09:32:38 +10:00 · 2022-04-05 09:32:38 +10:00 · 2022-04-05 09:32:38 +10:00 · 2021-12-13 12:26:14 +01:00
12 changed files with 171 additions and 184 deletions
--- a/.github/workflows/automerge.yml
+++ b/.github/workflows/automerge.yml
@@ -1,53 +1,11 @@
 # File managed by web3-bot. DO NOT EDIT.
 # See https://github.com/protocol/.github/ for details.

-# Automatically merge pull requests opened by web3-bot, as soon as (and only if) all tests pass.
-# This reduces the friction associated with updating with our workflows.
-
-on: [ pull_request ]
 name: Automerge
+on: [ pull_request ]

 jobs:
-  automerge-check:
-    if: github.event.pull_request.user.login == 'web3-bot'
-    runs-on: ubuntu-latest
-    outputs:
-      status: ${{ steps.should-automerge.outputs.status }}
-    steps:
-    - uses: actions/checkout@v2
-      with:
-        fetch-depth: 0
-    - name: Check if we should automerge
-      id: should-automerge
-      run: |
-        for commit in $(git rev-list --first-parent origin/${{ github.event.pull_request.base.ref }}..${{ github.event.pull_request.head.sha }}); do
-          committer=$(git show --format=$'%ce' -s $commit)
-          echo "Committer: $committer"
-          if [[ "$committer" != "web3-bot@users.noreply.github.com" ]]; then
-            echo "Commit $commit wasn't committed by web3-bot, but by $committer."
-            echo "::set-output name=status::false"
-            exit
-          fi
-        done
-        echo "::set-output name=status::true"
  automerge:
-    needs: automerge-check
-    runs-on: ubuntu-latest
-    # The check for the user is redundant here, as this job depends on the automerge-check job,
-    # but it prevents this job from spinning up, just to be skipped shortly after.
-    if: github.event.pull_request.user.login == 'web3-bot' && needs.automerge-check.outputs.status == 'true'
-    steps:
-    - name: Wait on tests
-      uses: lewagon/wait-on-check-action@bafe56a6863672c681c3cf671f5e10b20abf2eaa # v0.2
-      with:
-        ref: ${{ github.event.pull_request.head.sha }}
-        repo-token: ${{ secrets.GITHUB_TOKEN }}
-        wait-interval: 10
-        running-workflow-name: 'automerge' # the name of this job
-    - name: Merge PR
-      uses: pascalgn/automerge-action@741c311a47881be9625932b0a0de1b0937aab1ae # v0.13.1
-      env:
-        GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
-        MERGE_LABELS: ""
-        MERGE_METHOD: "squash"
-        MERGE_DELETE_BRANCH: true
+    uses: protocol/.github/.github/workflows/automerge.yml@master
+    with:
+      job: 'automerge'
--- a/.github/workflows/go-check.yml
+++ b/.github/workflows/go-check.yml
@@ -16,7 +16,7 @@ jobs:
          submodules: recursive
      - uses: actions/setup-go@v2
        with:
-          go-version: "1.17.x"
+          go-version: "1.18.x"
      - name: Run repo-specific setup
        uses: ./.github/actions/go-check-setup
        if: hashFiles('./.github/actions/go-check-setup') != ''
@@ -27,7 +27,7 @@ jobs:
            echo "RUNGOGENERATE=true" >> $GITHUB_ENV
          fi
      - name: Install staticcheck
-        run: go install honnef.co/go/tools/cmd/staticcheck@df71e5d0e0ed317ebf43e6e59cf919430fa4b8f2 # 2021.1.1 (v0.2.1)
+        run: go install honnef.co/go/tools/cmd/staticcheck@d7e217c1ff411395475b2971c0824e1e7cc1af98 # 2022.1 (v0.3.0)
      - name: Check that go.mod is tidy
        uses: protocol/multiple-go-modules@v1.2
        with:
@@ -71,4 +71,3 @@ jobs:
              git status --short
              exit 1
            fi
-
--- a/.github/workflows/go-test.yml
+++ b/.github/workflows/go-test.yml
@@ -10,11 +10,11 @@ jobs:
      fail-fast: false
      matrix:
        os: [ "ubuntu", "windows", "macos" ]
-        go: [ "1.16.x", "1.17.x" ]
+        go: [ "1.17.x", "1.18.x" ]
    env:
      COVERAGES: ""
-    runs-on: ${{ matrix.os }}-latest
-    name: ${{ matrix.os}} (go ${{ matrix.go }})
+    runs-on: ${{ format('{0}-latest', matrix.os) }}
+    name: ${{ matrix.os }} (go ${{ matrix.go }})
    steps:
      - uses: actions/checkout@v2
        with:
@@ -26,20 +26,33 @@ jobs:
        run: |
          go version
          go env
+      - name: Use msys2 on windows
+        if: ${{ matrix.os == 'windows' }}
+        shell: bash
+        # The executable for msys2 is also called bash.cmd
+        #   https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#shells
+        # If we prepend its location to the PATH
+        #   subsequent 'shell: bash' steps will use msys2 instead of gitbash
+        run: echo "C:/msys64/usr/bin" >> $GITHUB_PATH
      - name: Run repo-specific setup
        uses: ./.github/actions/go-test-setup
        if: hashFiles('./.github/actions/go-test-setup') != ''
      - name: Run tests
        uses: protocol/multiple-go-modules@v1.2
        with:
-          run: go test -v -coverprofile module-coverage.txt ./...
+          # Use -coverpkg=./..., so that we include cross-package coverage.
+          # If package ./A imports ./B, and ./A's tests also cover ./B,
+          # this means ./B's coverage will be significantly higher than 0%.
+          run: go test -v -coverprofile=module-coverage.txt -coverpkg=./... ./...
      - name: Run tests (32 bit)
        if: ${{ matrix.os != 'macos' }} # can't run 32 bit tests on OSX.
        uses: protocol/multiple-go-modules@v1.2
        env:
          GOARCH: 386
        with:
-          run: go test -v ./...
+          run: |
+            export "PATH=${{ env.PATH_386 }}:$PATH"
+            go test -v ./...
      - name: Run tests with race detector
        if: ${{ matrix.os == 'ubuntu' }} # speed things up. Windows and OSX VMs are slow
        uses: protocol/multiple-go-modules@v1.2
@@ -49,7 +62,7 @@ jobs:
        shell: bash
        run: echo "COVERAGES=$(find . -type f -name 'module-coverage.txt' | tr -s '\n' ',' | sed 's/,$//')" >> $GITHUB_ENV
      - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@51d810878be5422784e86451c0e7c14e5860ec47 # v2.0.2
+        uses: codecov/codecov-action@f32b3a3741e1053eb607407145bc9619351dc93b # v2.1.0
        with:
          files: '${{ env.COVERAGES }}'
          env_vars: OS=${{ matrix.os }}, GO=${{ matrix.go }}
--- a/.github/workflows/release-check.yml
+++ b/.github/workflows/release-check.yml
@@ -0,0 +1,11 @@
+# File managed by web3-bot. DO NOT EDIT.
+# See https://github.com/protocol/.github/ for details.
+
+name: Release Checker
+on:
+  pull_request:
+    paths: [ 'version.json' ]
+
+jobs:
+  release-check:
+    uses: protocol/.github/.github/workflows/release-check.yml@master
--- a/.github/workflows/releaser.yml
+++ b/.github/workflows/releaser.yml
@@ -0,0 +1,11 @@
+# File managed by web3-bot. DO NOT EDIT.
+# See https://github.com/protocol/.github/ for details.
+
+name: Releaser
+on:
+  push:
+    paths: [ 'version.json' ]
+
+jobs:
+  releaser:
+    uses: protocol/.github/.github/workflows/releaser.yml@master
--- a/.github/workflows/tagpush.yml
+++ b/.github/workflows/tagpush.yml
@@ -0,0 +1,12 @@
+# File managed by web3-bot. DO NOT EDIT.
+# See https://github.com/protocol/.github/ for details.
+
+name: Tag Push Checker
+on:
+  push:
+    tags:
+      - v*
+
+jobs:
+  releaser:
+    uses: protocol/.github/.github/workflows/tagpush.yml@master
--- a/README.md
+++ b/README.md
@@ -59,10 +59,17 @@ fmt.Println("Got CID: ", c)
 #### Creating a CID from scratch

 ```go
+
+import (
+  cid "github.com/ipfs/go-cid"
+  mc "github.com/multiformats/go-multicodec"
+  mh "github.com/multiformats/go-multihash"
+)
+
 // Create a cid manually by specifying the 'prefix' parameters
 pref := cid.Prefix{
 	Version: 1,
-	Codec: cid.Raw,
+	Codec: mc.Raw,
 	MhType: mh.SHA2_256,
 	MhLength: -1, // default length
 }
--- a/benchmark_test.go
+++ b/benchmark_test.go
@@ -0,0 +1,61 @@
+package cid_test
+
+import (
+	"math/rand"
+	"testing"
+
+	"github.com/ipfs/go-cid"
+	"github.com/multiformats/go-multihash"
+)
+
+// BenchmarkIdentityCheck benchmarks two ways of checking whether a CIDv1 has multihash.IDENTITY
+// code:
+//  1. Cid.Prefix(), and
+//  2. decoding the Cid.Hash().
+//
+// This benchmark illustrates that using Cid.Prefix is more efficient than multihash.Decode.
+// Users wishing to perform such a check should use Cid.Prefix.
+//
+// Consider that `Cid.Prefix` is already efficient enough and introducing a dedicated API for
+// performing this check will likely result in small gains.
+func BenchmarkIdentityCheck(b *testing.B) {
+	rng := rand.New(rand.NewSource(1413))
+
+	data := make([]byte, rng.Intn(100)+1024)
+	if _, err := rng.Read(data); err != nil {
+		b.Fatal(err)
+	}
+	mh, err := multihash.Sum(data, multihash.IDENTITY, -1)
+	if err != nil {
+		b.Fatal(err)
+	}
+	cv1 := cid.NewCidV1(cid.Raw, mh)
+
+	b.SetBytes(int64(cv1.ByteLen()))
+	b.ReportAllocs()
+	b.ResetTimer()
+
+	b.Run("Prefix", func(b *testing.B) {
+		b.RunParallel(func(pb *testing.PB) {
+			for pb.Next() {
+				if cv1.Prefix().MhType != multihash.IDENTITY {
+					b.Fatal("expected IDENTITY CID")
+				}
+			}
+		})
+	})
+
+	b.Run("MultihashDecode", func(b *testing.B) {
+		b.RunParallel(func(pb *testing.PB) {
+			for pb.Next() {
+				dmh, err := multihash.Decode(cv1.Hash())
+				if err != nil {
+					b.Fatal(err)
+				}
+				if dmh.Code != multihash.IDENTITY {
+					b.Fatal("expected IDENTITY CID")
+				}
+			}
+		})
+	})
+}
--- a/cid.go
+++ b/cid.go
@@ -22,6 +22,7 @@ package cid
 import (
 	"bytes"
 	"encoding"
+	"encoding/binary"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -46,19 +47,21 @@ var (
 	ErrInvalidEncoding = errors.New("invalid base encoding")
 )

-// These are multicodec-packed content types. The should match
-// the codes described in the authoritative document:
-// https://github.com/multiformats/multicodec/blob/master/table.csv
+// Consts below are DEPRECATED and left only for legacy reasons:
+// <https://github.com/ipfs/go-cid/pull/137>
+// Modern code should use consts from go-multicodec instead:
+// <https://github.com/multiformats/go-multicodec>
 const (
-	Raw = 0x55
+	// common ones
+	Raw         = 0x55
+	DagProtobuf = 0x70   // https://ipld.io/docs/codecs/known/dag-pb/
+	DagCBOR     = 0x71   // https://ipld.io/docs/codecs/known/dag-cbor/
+	DagJSON     = 0x0129 // https://ipld.io/docs/codecs/known/dag-json/
+	Libp2pKey   = 0x72   // https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#peer-ids

-	DagProtobuf = 0x70
-	DagCBOR     = 0x71
-	Libp2pKey   = 0x72
-
-	GitRaw = 0x78
-
-	DagJOSE               = 0x85
+	// other
+	GitRaw                = 0x78
+	DagJOSE               = 0x85 // https://ipld.io/specs/codecs/dag-jose/spec/
 	EthBlock              = 0x90
 	EthBlockList          = 0x91
 	EthTxTrie             = 0x92
@@ -80,64 +83,6 @@ const (
 	FilCommitmentSealed   = 0xf102
 )

-// Codecs maps the name of a codec to its type
-var Codecs = map[string]uint64{
-	"v0":                      DagProtobuf,
-	"raw":                     Raw,
-	"protobuf":                DagProtobuf,
-	"cbor":                    DagCBOR,
-	"libp2p-key":              Libp2pKey,
-	"git-raw":                 GitRaw,
-	"eth-block":               EthBlock,
-	"eth-block-list":          EthBlockList,
-	"eth-tx-trie":             EthTxTrie,
-	"eth-tx":                  EthTx,
-	"eth-tx-receipt-trie":     EthTxReceiptTrie,
-	"eth-tx-receipt":          EthTxReceipt,
-	"eth-state-trie":          EthStateTrie,
-	"eth-account-snapshot":    EthAccountSnapshot,
-	"eth-storage-trie":        EthStorageTrie,
-	"bitcoin-block":           BitcoinBlock,
-	"bitcoin-tx":              BitcoinTx,
-	"zcash-block":             ZcashBlock,
-	"zcash-tx":                ZcashTx,
-	"decred-block":            DecredBlock,
-	"decred-tx":               DecredTx,
-	"dash-block":              DashBlock,
-	"dash-tx":                 DashTx,
-	"fil-commitment-unsealed": FilCommitmentUnsealed,
-	"fil-commitment-sealed":   FilCommitmentSealed,
-	"dag-jose":                DagJOSE,
-}
-
-// CodecToStr maps the numeric codec to its name
-var CodecToStr = map[uint64]string{
-	Raw:                   "raw",
-	DagProtobuf:           "protobuf",
-	DagCBOR:               "cbor",
-	GitRaw:                "git-raw",
-	EthBlock:              "eth-block",
-	EthBlockList:          "eth-block-list",
-	EthTxTrie:             "eth-tx-trie",
-	EthTx:                 "eth-tx",
-	EthTxReceiptTrie:      "eth-tx-receipt-trie",
-	EthTxReceipt:          "eth-tx-receipt",
-	EthStateTrie:          "eth-state-trie",
-	EthAccountSnapshot:    "eth-account-snapshot",
-	EthStorageTrie:        "eth-storage-trie",
-	BitcoinBlock:          "bitcoin-block",
-	BitcoinTx:             "bitcoin-tx",
-	ZcashBlock:            "zcash-block",
-	ZcashTx:               "zcash-tx",
-	DecredBlock:           "decred-block",
-	DecredTx:              "decred-tx",
-	DashBlock:             "dash-block",
-	DashTx:                "dash-tx",
-	FilCommitmentUnsealed: "fil-commitment-unsealed",
-	FilCommitmentSealed:   "fil-commitment-sealed",
-	DagJOSE:               "dag-jose",
-}
-
 // tryNewCidV0 tries to convert a multihash into a CIDv0 CID and returns an
 // error on failure.
 func tryNewCidV0(mhash mh.Multihash) (Cid, error) {
@@ -173,16 +118,24 @@ func NewCidV0(mhash mh.Multihash) Cid {
 // Panics if the multihash is invalid.
 func NewCidV1(codecType uint64, mhash mh.Multihash) Cid {
 	hashlen := len(mhash)
-	// two 8 bytes (max) numbers plus hash
-	buf := make([]byte, 1+varint.UvarintSize(codecType)+hashlen)
-	n := varint.PutUvarint(buf, 1)
-	n += varint.PutUvarint(buf[n:], codecType)
-	cn := copy(buf[n:], mhash)
+
+	// Two 8 bytes (max) numbers plus hash.
+	// We use strings.Builder to only allocate once.
+	var b strings.Builder
+	b.Grow(1 + varint.UvarintSize(codecType) + hashlen)
+
+	b.WriteByte(1)
+
+	var buf [binary.MaxVarintLen64]byte
+	n := varint.PutUvarint(buf[:], codecType)
+	b.Write(buf[:n])
+
+	cn, _ := b.Write(mhash)
 	if cn != hashlen {
 		panic("copy hash length is inconsistent")
 	}

-	return Cid{string(buf[:n+hashlen])}
+	return Cid{b.String()}
 }

 var (
--- a/cid_test.go
+++ b/cid_test.go
@@ -15,37 +15,6 @@ import (
 	mh "github.com/multiformats/go-multihash"
 )

-// Copying the "silly test" idea from
-// https://github.com/multiformats/go-multihash/blob/7aa9f26a231c6f34f4e9fad52bf580fd36627285/multihash_test.go#L13
-// Makes it so changing the table accidentally has to happen twice.
-var tCodecs = map[uint64]string{
-	Raw:                   "raw",
-	DagProtobuf:           "protobuf",
-	DagCBOR:               "cbor",
-	Libp2pKey:             "libp2p-key",
-	GitRaw:                "git-raw",
-	EthBlock:              "eth-block",
-	EthBlockList:          "eth-block-list",
-	EthTxTrie:             "eth-tx-trie",
-	EthTx:                 "eth-tx",
-	EthTxReceiptTrie:      "eth-tx-receipt-trie",
-	EthTxReceipt:          "eth-tx-receipt",
-	EthStateTrie:          "eth-state-trie",
-	EthAccountSnapshot:    "eth-account-snapshot",
-	EthStorageTrie:        "eth-storage-trie",
-	BitcoinBlock:          "bitcoin-block",
-	BitcoinTx:             "bitcoin-tx",
-	ZcashBlock:            "zcash-block",
-	ZcashTx:               "zcash-tx",
-	DecredBlock:           "decred-block",
-	DecredTx:              "decred-tx",
-	DashBlock:             "dash-block",
-	DashTx:                "dash-tx",
-	FilCommitmentUnsealed: "fil-commitment-unsealed",
-	FilCommitmentSealed:   "fil-commitment-sealed",
-	DagJOSE:               "dag-jose",
-}
-
 func assertEqual(t *testing.T, a, b Cid) {
 	if a.Type() != b.Type() {
 		t.Fatal("mismatch on type")
@@ -60,26 +29,6 @@ func assertEqual(t *testing.T, a, b Cid) {
 	}
 }

-func TestTable(t *testing.T) {
-	if len(tCodecs) != len(Codecs)-1 {
-		t.Errorf("Item count mismatch in the Table of Codec. Should be %d, got %d", len(tCodecs)+1, len(Codecs))
-	}
-
-	for k, v := range tCodecs {
-		if Codecs[v] != k {
-			t.Errorf("Table mismatch: 0x%x %s", k, v)
-		}
-	}
-}
-
-// The table returns cid.DagProtobuf for "v0"
-// so we test it apart
-func TestTableForV0(t *testing.T) {
-	if Codecs["v0"] != DagProtobuf {
-		t.Error("Table mismatch: Codecs[\"v0\"] should resolve to DagProtobuf (0x70)")
-	}
-}
-
 func TestPrefixSum(t *testing.T) {
 	// Test creating CIDs both manually and with Prefix.
 	// Tests: https://github.com/ipfs/go-cid/issues/83
--- a/go.mod
+++ b/go.mod
@@ -4,7 +4,17 @@ require (
 	github.com/multiformats/go-multibase v0.0.3
 	github.com/multiformats/go-multihash v0.0.15
 	github.com/multiformats/go-varint v0.0.6
-	golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf // indirect
 )

-go 1.16
+require (
+	github.com/klauspost/cpuid/v2 v2.0.4 // indirect
+	github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 // indirect
+	github.com/minio/sha256-simd v1.0.0 // indirect
+	github.com/mr-tron/base58 v1.2.0 // indirect
+	github.com/multiformats/go-base32 v0.0.3 // indirect
+	github.com/multiformats/go-base36 v0.1.0 // indirect
+	golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf // indirect
+	golang.org/x/sys v0.0.0-20210309074719-68d13333faf2 // indirect
+)
+
+go 1.17
--- a/version.json
+++ b/version.json
@@ -0,0 +1,3 @@
+{
+  "version": "v0.2.0"
+}
Author	SHA1	Message	Date
Marcin Rataj	b2064d74a8	fix: remove invalid multicodec2string mappings (#137 ) * refactor: remove Codecs table * chore: go-cid 0.2.0 Codec table was missing dag-json and it had invalid code for dag-cbor. It also had invalid string representation of dag-pb -- it was using 'protobuf' which is a totally different code. This removes invalid mappings entirely. https://github.com/multiformats/go-multicodec should be used instead.	2022-04-21 22:39:31 +02:00
web3-bot	ddd9ef7248	update .github/workflows/go-check.yml	2022-04-05 09:32:38 +10:00
web3-bot	6be8b631d0	update .github/workflows/go-test.yml	2022-04-05 09:32:38 +10:00
web3-bot	395d392889	update .github/workflows/automerge.yml	2022-04-05 09:32:38 +10:00
web3-bot	1d11990417	bump go.mod to Go 1.17 and run go fix	2022-04-05 09:32:38 +10:00
web3-bot	dc3bb41324	sync: update CI config files (#136 ) * add version.json file * update .github/workflows/go-test.yml * update .github/workflows/go-check.yml * add .github/workflows/releaser.yml * add .github/workflows/release-check.yml * add .github/workflows/tagpush.yml Co-authored-by: web3-bot <web3-bot@users.noreply.github.com>	2021-12-13 12:26:14 +01:00
Masih H. Derkani	28f4a5eab6	Benchmark existing ways to check for `IDENTITY` CIDs Add benchmarks that compare two ways of checking for `multihash.IDENTITY` code: 1. `Cid.Prefix().MhType` 2. Decode of `Cid.Hash()` This benchmark illustrates that using Cid.Prefix is more efficient than `multihash.Decode`. Users wishing to perform such a check should use `Cid.Prefix`. Consider that `Cid.Prefix` is already efficient enough and introducing a dedicated API for performing this check will likely result in small gains. Relates to #133	2021-11-08 14:52:45 +00:00
Daniel Martí	5640b0169f	avoid double alloc in NewCidV1 We allocate once via "make([]byte, len)", and again when that buffer is converted to a string. Thankfully, since Go 1.10 we have strings.Builder, designed specifically for this use case. In a downstream benchmark in go-car, which needs to reconstruct many CID values, we see small but nice gains: name old time/op new time/op delta ReadBlocks-16 1.09ms ± 4% 1.06ms ± 5% -3.33% (p=0.007 n=11+11) name old speed new speed delta ReadBlocks-16 478MB/s ± 4% 494MB/s ± 5% +3.46% (p=0.007 n=11+11) name old alloc/op new alloc/op delta ReadBlocks-16 1.30MB ± 0% 1.25MB ± 0% -3.86% (p=0.000 n=12+12) name old allocs/op new allocs/op delta ReadBlocks-16 9.50k ± 0% 8.45k ± 0% -11.05% (p=0.000 n=12+12)	2021-09-12 14:08:59 +02:00