gx publish 0.9.0

Create a new Encode method that is like StringOfBase but never errors

.gx/lastpubver

@@ -1 +1 @@
-0.7.19: QmeSrf6pzut73u6zLQkRFQ3ygt3k6XFT2kjdYP8Tnkwwyg
+0.9.0: QmPSQnBKM9g7BaUcZCvswUJVscQ1ipjmwxN5PXCjkp9EQ7

_rsrch/cidiface/README.md

@@ -0,0 +1,168 @@
+What golang Kinds work best to implement CIDs?
+==============================================
+
+There are many possible ways to implement CIDs.  This package explores them.
+
+### criteria
+
+There's a couple different criteria to consider:
+
+- We want the best performance when operating on the type (getters, mostly);
+- We want to minimize the number of memory allocations we need;
+- We want types which can be used as map keys, because this is common.
+
+The priority of these criteria is open to argument, but it's probably
+mapkeys > minalloc > anythingelse.
+(Mapkeys and minalloc are also quite entangled, since if we don't pick a
+representation that can work natively as a map key, we'll end up needing
+a `KeyRepr()` method which gives us something that does work as a map key,
+an that will almost certainly involve a malloc itself.)
+
+### options
+
+There are quite a few different ways to go:
+
+- Option A: CIDs as a struct; multihash as bytes.
+- Option B: CIDs as a string.
+- Option C: CIDs as an interface with multiple implementors.
+- Option D: CIDs as a struct; multihash also as a struct or string.
+- Option E: CIDs as a struct; content as strings plus offsets.
+- Option F: CIDs as a struct wrapping only a string.
+
+The current approach on the master branch is Option A.
+
+Option D is distinctive from Option A because multihash as bytes transitively
+causes the CID struct to be non-comparible and thus not suitable for map keys
+as per https://golang.org/ref/spec#KeyType .  (It's also a bit more work to
+pursue Option D because it's just a bigger splash radius of change; but also,
+something we might also want to do soon, because we *do* also have these same
+map-key-usability concerns with multihash alone.)
+
+Option E is distinctive from Option D because Option E would always maintain
+the binary format of the cid internally, and so could yield it again without
+malloc, while still potentially having faster access to components than
+Option B since it wouldn't need to re-parse varints to access later fields.
+
+Option F is actually a varation of Option B; it's distinctive from the other
+struct options because it is proposing *literally* `struct{ x string }` as
+the type, with no additional fields for components nor offsets.
+
+Option C is the avoid-choices choice, but note that interfaces are not free;
+since "minimize mallocs" is one of our major goals, we cannot use interfaces
+whimsically.
+
+Note there is no proposal for migrating to `type Cid []bytes`, because that
+is generally considered to be strictly inferior to `type Cid string`.
+
+
+Discoveries
+-----------
+
+### using interfaces as map keys forgoes a lot of safety checks
+
+Using interfaces as map keys pushes a bunch of type checking to runtime.
+E.g., it's totally valid at compile time to push a type which is non-comparable
+into a map key; it will panic at *runtime* instead of failing at compile-time.
+
+There's also no way to define equality checks between implementors of the
+interface: golang will always use its innate concept of comparison for the
+concrete types.  This means its effectively *never safe* to use two different
+concrete implementations of an interface in the same map; you may add elements
+which are semantically "equal" in your mind, and end up very confused later
+when both impls of the same "equal" object have been stored.
+
+### sentinel values are possible in any impl, but some are clearer than others
+
+When using `*Cid`, the nil value is a clear sentinel for 'invalid';
+when using `type Cid string`, the zero value is a clear sentinel;
+when using `type Cid struct` per Option A or D... the only valid check is
+for a nil multihash field, since version=0 and codec=0 are both valid values.
+When using `type Cid struct{string}` per Option F, zero is a clear sentinel.
+
+### usability as a map key is important
+
+We already covered this in the criteria section, but for clarity:
+
+- Option A: ❌
+- Option B: ✔
+- Option C: ~ (caveats, and depends on concrete impl)
+- Option D: ✔
+- Option E: ✔
+- Option F: ✔
+
+### living without offsets requires parsing
+
+Since CID (and multihash!) are defined using varints, they require parsing;
+we can't just jump into the string at a known offset in order to yield e.g.
+the multicodec number.
+
+In order to get to the 'meat' of the CID (the multihash content), we first
+must parse:
+
+- the CID version varint;
+- the multicodec varint;
+- the multihash type enum varint;
+- and the multihash length varint.
+
+Since there are many applications where we want to jump straight to the
+multihash content (for example, when doing CAS sharding -- see the
+[disclaimer](https://github.com/multiformats/multihash#disclaimers) about
+bias in leading bytes), this overhead may be interesting.
+
+How much this overhead is significant is hard to say from microbenchmarking;
+it depends largely on usage patterns. If these traversals are a significant
+timesink, it would be an argument for Option D/E.
+If these traversals are *not* a significant timesink, we might be wiser
+to keep to Option B/F, because keeping a struct full of offsets will add several
+words of memory usage per CID, and we keep a *lot* of CIDs.
+
+### interfaces cause boxing which is a significant performance cost
+
+See `BenchmarkCidMap_CidStr` and friends.
+
+Long story short: using interfaces *anywhere* will cause the compiler to
+implicitly generate boxing and unboxing code (e.g. `runtime.convT2E`);
+this is both another function call, and more concerningly, results in
+large numbers of unbatchable memory allocations.
+
+Numbers without context are dangerous, but if you need one: 33%.
+It's a big deal.
+
+This means attempts to "use interfaces, but switch to concrete impls when
+performance is important" are a red herring: it doesn't work that way.
+
+This is not a general inditement against using interfaces -- but
+if a situation is at the scale where it's become important to mind whether
+or not pointers are a performance impact, then that situation also
+is one where you have to think twice before using interfaces.
+
+### struct wrappers can be used in place of typedefs with zero overhead
+
+See `TestSizeOf`.
+
+Using the `unsafe.Sizeof` feature to inspect what the Go runtime thinks,
+we can see that `type Foo string` and `type Foo struct{x string}` consume
+precisely the same amount of memory.
+
+This is interesting because it means we can choose between either
+type definition with no significant overhead anywhere we use it:
+thus, we can choose freely between Option B and Option F based on which
+we feel is more pleasant to work with.
+
+Option F (a struct wrapper) means we can prevent casting into our Cid type.
+Option B (typedef string) can be declared a `const`.
+Are there any other concerns that would separate the two choices?
+
+### one way or another: let's get rid of that star
+
+We should switch completely to handling `Cid` and remove `*Cid` completely.
+Regardless of whether we do this by migrating to interface, or string
+implementations, or simply structs with no pointers... once we get there,
+refactoring to any of the *others* can become a no-op from the perspective
+of any downstream code that uses CIDs.
+
+(This means all access via functions, never references to fields -- even if
+we were to use a struct implementation.  *Pretend* there's a interface,
+in other words.)
+
+There are probably `gofix` incantations which can help us with this migration.

_rsrch/cidiface/cid.go

@@ -0,0 +1,48 @@
+package cid
+
+import (
+	mh "github.com/multiformats/go-multihash"
+)
+
+// Cid represents a self-describing content adressed identifier.
+//
+// A CID is composed of:
+//
+//   - a Version of the CID itself,
+//   - a Multicodec (indicates the encoding of the referenced content),
+//   - and a Multihash (which identifies the referenced content).
+//
+// (Note that the Multihash further contains its own version and hash type
+// indicators.)
+type Cid interface {
+	// n.b. 'yields' means "without copy", 'produces' means a malloc.
+
+	Version() uint64         // Yields the version prefix as a uint.
+	Multicodec() uint64      // Yields the multicodec as a uint.
+	Multihash() mh.Multihash // Yields the multihash segment.
+
+	String() string // Produces the CID formatted as b58 string.
+	Bytes() []byte  // Produces the CID formatted as raw binary.
+
+	Prefix() Prefix // Produces a tuple of non-content metadata.
+
+	// some change notes:
+	// - `KeyString() CidString` is gone because we're natively a map key now, you're welcome.
+	// - `StringOfBase(mbase.Encoding) (string, error)` is skipped, maybe it can come back but maybe it should be a formatter's job.
+	// - `Equals(o Cid) bool` is gone because it's now `==`, you're welcome.
+
+	// TODO: make a multi-return method for {v,mc,mh} decomposition.  CidStr will be able to implement this more efficiently than if one makes a series of the individual getter calls.
+}
+
+// Prefix represents all the metadata of a Cid,
+// that is, the Version, the Codec, the Multihash type
+// and the Multihash length. It does not contains
+// any actual content information.
+// NOTE: The use -1 in MhLength to mean default length is deprecated,
+//   use the V0Builder or V1Builder structures instead
+type Prefix struct {
+	Version  uint64
+	Codec    uint64
+	MhType   uint64
+	MhLength int
+}

_rsrch/cidiface/cidBoxingBench_test.go

@@ -0,0 +1,71 @@
+package cid
+
+import (
+	"testing"
+)
+
+// BenchmarkCidMap_CidStr estimates how fast it is to insert primitives into a map
+// keyed by CidStr (concretely).
+//
+// We do 100 insertions per benchmark run to make sure the map initialization
+// doesn't dominate the results.
+//
+// Sample results on linux amd64 go1.11beta:
+//
+//   BenchmarkCidMap_CidStr-8          100000             16317 ns/op
+//   BenchmarkCidMap_CidIface-8        100000             20516 ns/op
+//
+// With benchmem on:
+//
+//   BenchmarkCidMap_CidStr-8          100000             15579 ns/op           11223 B/op        207 allocs/op
+//   BenchmarkCidMap_CidIface-8        100000             19500 ns/op           12824 B/op        307 allocs/op
+//   BenchmarkCidMap_StrPlusHax-8      200000             10451 ns/op            7589 B/op        202 allocs/op
+//
+// We can see here that the impact of interface boxing is significant:
+// it increases the time taken to do the inserts to 133%, largely because
+// the implied `runtime.convT2E` calls cause another malloc each.
+//
+// There are also significant allocations in both cases because
+// A) we cannot create a multihash without allocations since they are []byte;
+// B) the map has to be grown several times;
+// C) something I haven't quite put my finger on yet.
+// Ideally we'd drive those down further as well.
+//
+// Pre-allocating the map reduces allocs by a very small percentage by *count*,
+// but reduces the time taken by 66% overall (presumably because when a map
+// re-arranges itself, it involves more or less an O(n) copy of the content
+// in addition to the alloc itself).  This isn't topical to the question of
+// whether or not interfaces are a good idea; just for contextualizing.
+//
+func BenchmarkCidMap_CidStr(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		mp := map[CidStr]int{}
+		for x := 0; x < 100; x++ {
+			mp[NewCidStr(0, uint64(x), []byte{})] = x
+		}
+	}
+}
+
+// BenchmarkCidMap_CidIface is in the family of BenchmarkCidMap_CidStr:
+// it is identical except the map key type is declared as an interface
+// (which forces all insertions to be boxed, changing performance).
+func BenchmarkCidMap_CidIface(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		mp := map[Cid]int{}
+		for x := 0; x < 100; x++ {
+			mp[NewCidStr(0, uint64(x), []byte{})] = x
+		}
+	}
+}
+
+// BenchmarkCidMap_CidStrAvoidMapGrowth is in the family of BenchmarkCidMap_CidStr:
+// it is identical except the map is created with a size hint that removes
+// some allocations (5, in practice, apparently).
+func BenchmarkCidMap_CidStrAvoidMapGrowth(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		mp := make(map[CidStr]int, 100)
+		for x := 0; x < 100; x++ {
+			mp[NewCidStr(0, uint64(x), []byte{})] = x
+		}
+	}
+}

_rsrch/cidiface/cidString.go

@@ -0,0 +1,161 @@
+package cid
+
+import (
+	"encoding/binary"
+	"fmt"
+
+	mbase "github.com/multiformats/go-multibase"
+	mh "github.com/multiformats/go-multihash"
+)
+
+//=================
+// def & accessors
+//=================
+
+var _ Cid = CidStr("")
+var _ map[CidStr]struct{} = nil
+
+// CidStr is a representation of a Cid as a string type containing binary.
+//
+// Using golang's string type is preferable over byte slices even for binary
+// data because golang strings are immutable, usable as map keys,
+// trivially comparable with built-in equals operators, etc.
+//
+// Please do not cast strings or bytes into the CidStr type directly;
+// use a parse method which validates the data and yields a CidStr.
+type CidStr string
+
+// EmptyCidStr is a constant for a zero/uninitialized/sentinelvalue cid;
+// it is declared mainly for readability in checks for sentinel values.
+const EmptyCidStr = CidStr("")
+
+func (c CidStr) Version() uint64 {
+	bytes := []byte(c)
+	v, _ := binary.Uvarint(bytes)
+	return v
+}
+
+func (c CidStr) Multicodec() uint64 {
+	bytes := []byte(c)
+	_, n := binary.Uvarint(bytes) // skip version length
+	codec, _ := binary.Uvarint(bytes[n:])
+	return codec
+}
+
+func (c CidStr) Multihash() mh.Multihash {
+	bytes := []byte(c)
+	_, n1 := binary.Uvarint(bytes)      // skip version length
+	_, n2 := binary.Uvarint(bytes[n1:]) // skip codec length
+	return mh.Multihash(bytes[n1+n2:])  // return slice of remainder
+}
+
+// String returns the default string representation of a Cid.
+// Currently, Base58 is used as the encoding for the multibase string.
+func (c CidStr) String() string {
+	switch c.Version() {
+	case 0:
+		return c.Multihash().B58String()
+	case 1:
+		mbstr, err := mbase.Encode(mbase.Base58BTC, []byte(c))
+		if err != nil {
+			panic("should not error with hardcoded mbase: " + err.Error())
+		}
+		return mbstr
+	default:
+		panic("not possible to reach this point")
+	}
+}
+
+// Bytes produces a raw binary format of the CID.
+//
+// (For CidStr, this method is only distinct from casting because of
+// compatibility with v0 CIDs.)
+func (c CidStr) Bytes() []byte {
+	switch c.Version() {
+	case 0:
+		return c.Multihash()
+	case 1:
+		return []byte(c)
+	default:
+		panic("not possible to reach this point")
+	}
+}
+
+// Prefix builds and returns a Prefix out of a Cid.
+func (c CidStr) Prefix() Prefix {
+	dec, _ := mh.Decode(c.Multihash()) // assuming we got a valid multiaddr, this will not error
+	return Prefix{
+		MhType:   dec.Code,
+		MhLength: dec.Length,
+		Version:  c.Version(),
+		Codec:    c.Multicodec(),
+	}
+}
+
+//==================================
+// parsers & validators & factories
+//==================================
+
+func NewCidStr(version uint64, codecType uint64, mhash mh.Multihash) CidStr {
+	hashlen := len(mhash)
+	// two 8 bytes (max) numbers plus hash
+	buf := make([]byte, 2*binary.MaxVarintLen64+hashlen)
+	n := binary.PutUvarint(buf, version)
+	n += binary.PutUvarint(buf[n:], codecType)
+	cn := copy(buf[n:], mhash)
+	if cn != hashlen {
+		panic("copy hash length is inconsistent")
+	}
+	return CidStr(buf[:n+hashlen])
+}
+
+// CidStrParse takes a binary byte slice, parses it, and returns either
+// a valid CidStr, or the zero CidStr and an error.
+//
+// For CidV1, the data buffer is in the form:
+//
+//     <version><codec-type><multihash>
+//
+// CidV0 are also supported. In particular, data buffers starting
+// with length 34 bytes, which starts with bytes [18,32...] are considered
+// binary multihashes.
+//
+// The multicodec bytes are not parsed to verify they're a valid varint;
+// no further reification is performed.
+//
+// Multibase encoding should already have been unwrapped before parsing;
+// if you have a multibase-enveloped string, use CidStrDecode instead.
+//
+// CidStrParse is the inverse of Cid.Bytes().
+func CidStrParse(data []byte) (CidStr, error) {
+	if len(data) == 34 && data[0] == 18 && data[1] == 32 {
+		h, err := mh.Cast(data)
+		if err != nil {
+			return EmptyCidStr, err
+		}
+		return NewCidStr(0, DagProtobuf, h), nil
+	}
+
+	vers, n := binary.Uvarint(data)
+	if err := uvError(n); err != nil {
+		return EmptyCidStr, err
+	}
+
+	if vers != 0 && vers != 1 {
+		return EmptyCidStr, fmt.Errorf("invalid cid version number: %d", vers)
+	}
+
+	_, cn := binary.Uvarint(data[n:])
+	if err := uvError(cn); err != nil {
+		return EmptyCidStr, err
+	}
+
+	rest := data[n+cn:]
+	h, err := mh.Cast(rest)
+	if err != nil {
+		return EmptyCidStr, err
+	}
+
+	// REVIEW: if the data is longer than the mh.len expects, we silently ignore it?  should we?
+	return CidStr(data[0 : n+cn+len(h)]), nil
+}

_rsrch/cidiface/cidStruct.go

@@ -0,0 +1,164 @@
+package cid
+
+import (
+	"encoding/binary"
+	"fmt"
+
+	mbase "github.com/multiformats/go-multibase"
+	mh "github.com/multiformats/go-multihash"
+)
+
+//=================
+// def & accessors
+//=================
+
+var _ Cid = CidStruct{}
+
+//var _ map[CidStruct]struct{} = nil // Will not compile!  See struct def docs.
+//var _ map[Cid]struct{} = map[Cid]struct{}{CidStruct{}: struct{}{}} // Legal to compile...
+// but you'll get panics: "runtime error: hash of unhashable type cid.CidStruct"
+
+// CidStruct represents a CID in a struct format.
+//
+// This format complies with the exact same Cid interface as the CidStr
+// implementation, but completely pre-parses the Cid metadata.
+// CidStruct is a tad quicker in case of repeatedly accessed fields,
+// but requires more reshuffling to parse and to serialize.
+// CidStruct is not usable as a map key, because it contains a Multihash
+// reference, which is a slice, and thus not "comparable" as a primitive.
+//
+// Beware of zero-valued CidStruct: it is difficult to distinguish an
+// incorrectly-initialized "invalid" CidStruct from one representing a v0 cid.
+type CidStruct struct {
+	version uint64
+	codec   uint64
+	hash    mh.Multihash
+}
+
+// EmptyCidStruct is a constant for a zero/uninitialized/sentinelvalue cid;
+// it is declared mainly for readability in checks for sentinel values.
+//
+// Note: it's not actually a const; the compiler does not allow const structs.
+var EmptyCidStruct = CidStruct{}
+
+func (c CidStruct) Version() uint64 {
+	return c.version
+}
+
+func (c CidStruct) Multicodec() uint64 {
+	return c.codec
+}
+
+func (c CidStruct) Multihash() mh.Multihash {
+	return c.hash
+}
+
+// String returns the default string representation of a Cid.
+// Currently, Base58 is used as the encoding for the multibase string.
+func (c CidStruct) String() string {
+	switch c.Version() {
+	case 0:
+		return c.Multihash().B58String()
+	case 1:
+		mbstr, err := mbase.Encode(mbase.Base58BTC, c.Bytes())
+		if err != nil {
+			panic("should not error with hardcoded mbase: " + err.Error())
+		}
+		return mbstr
+	default:
+		panic("not possible to reach this point")
+	}
+}
+
+// Bytes produces a raw binary format of the CID.
+func (c CidStruct) Bytes() []byte {
+	switch c.version {
+	case 0:
+		return []byte(c.hash)
+	case 1:
+		// two 8 bytes (max) numbers plus hash
+		buf := make([]byte, 2*binary.MaxVarintLen64+len(c.hash))
+		n := binary.PutUvarint(buf, c.version)
+		n += binary.PutUvarint(buf[n:], c.codec)
+		cn := copy(buf[n:], c.hash)
+		if cn != len(c.hash) {
+			panic("copy hash length is inconsistent")
+		}
+		return buf[:n+len(c.hash)]
+	default:
+		panic("not possible to reach this point")
+	}
+}
+
+// Prefix builds and returns a Prefix out of a Cid.
+func (c CidStruct) Prefix() Prefix {
+	dec, _ := mh.Decode(c.hash) // assuming we got a valid multiaddr, this will not error
+	return Prefix{
+		MhType:   dec.Code,
+		MhLength: dec.Length,
+		Version:  c.version,
+		Codec:    c.codec,
+	}
+}
+
+//==================================
+// parsers & validators & factories
+//==================================
+
+// CidStructParse takes a binary byte slice, parses it, and returns either
+// a valid CidStruct, or the zero CidStruct and an error.
+//
+// For CidV1, the data buffer is in the form:
+//
+//     <version><codec-type><multihash>
+//
+// CidV0 are also supported. In particular, data buffers starting
+// with length 34 bytes, which starts with bytes [18,32...] are considered
+// binary multihashes.
+//
+// The multicodec bytes are not parsed to verify they're a valid varint;
+// no further reification is performed.
+//
+// Multibase encoding should already have been unwrapped before parsing;
+// if you have a multibase-enveloped string, use CidStructDecode instead.
+//
+// CidStructParse is the inverse of Cid.Bytes().
+func CidStructParse(data []byte) (CidStruct, error) {
+	if len(data) == 34 && data[0] == 18 && data[1] == 32 {
+		h, err := mh.Cast(data)
+		if err != nil {
+			return EmptyCidStruct, err
+		}
+		return CidStruct{
+			codec:   DagProtobuf,
+			version: 0,
+			hash:    h,
+		}, nil
+	}
+
+	vers, n := binary.Uvarint(data)
+	if err := uvError(n); err != nil {
+		return EmptyCidStruct, err
+	}
+
+	if vers != 0 && vers != 1 {
+		return EmptyCidStruct, fmt.Errorf("invalid cid version number: %d", vers)
+	}
+
+	codec, cn := binary.Uvarint(data[n:])
+	if err := uvError(cn); err != nil {
+		return EmptyCidStruct, err
+	}
+
+	rest := data[n+cn:]
+	h, err := mh.Cast(rest)
+	if err != nil {
+		return EmptyCidStruct, err
+	}
+
+	return CidStruct{
+		version: vers,
+		codec:   codec,
+		hash:    h,
+	}, nil
+}

_rsrch/cidiface/enums.go

@@ -0,0 +1,76 @@
+package cid
+
+// These are multicodec-packed content types. The should match
+// the codes described in the authoritative document:
+// https://github.com/multiformats/multicodec/blob/master/table.csv
+const (
+	Raw = 0x55
+
+	DagProtobuf = 0x70
+	DagCBOR     = 0x71
+
+	GitRaw = 0x78
+
+	EthBlock           = 0x90
+	EthBlockList       = 0x91
+	EthTxTrie          = 0x92
+	EthTx              = 0x93
+	EthTxReceiptTrie   = 0x94
+	EthTxReceipt       = 0x95
+	EthStateTrie       = 0x96
+	EthAccountSnapshot = 0x97
+	EthStorageTrie     = 0x98
+	BitcoinBlock       = 0xb0
+	BitcoinTx          = 0xb1
+	ZcashBlock         = 0xc0
+	ZcashTx            = 0xc1
+	DecredBlock        = 0xe0
+	DecredTx           = 0xe1
+)
+
+// Codecs maps the name of a codec to its type
+var Codecs = map[string]uint64{
+	"v0":                   DagProtobuf,
+	"raw":                  Raw,
+	"protobuf":             DagProtobuf,
+	"cbor":                 DagCBOR,
+	"git-raw":              GitRaw,
+	"eth-block":            EthBlock,
+	"eth-block-list":       EthBlockList,
+	"eth-tx-trie":          EthTxTrie,
+	"eth-tx":               EthTx,
+	"eth-tx-receipt-trie":  EthTxReceiptTrie,
+	"eth-tx-receipt":       EthTxReceipt,
+	"eth-state-trie":       EthStateTrie,
+	"eth-account-snapshot": EthAccountSnapshot,
+	"eth-storage-trie":     EthStorageTrie,
+	"bitcoin-block":        BitcoinBlock,
+	"bitcoin-tx":           BitcoinTx,
+	"zcash-block":          ZcashBlock,
+	"zcash-tx":             ZcashTx,
+	"decred-block":         DecredBlock,
+	"decred-tx":            DecredTx,
+}
+
+// CodecToStr maps the numeric codec to its name
+var CodecToStr = map[uint64]string{
+	Raw:                "raw",
+	DagProtobuf:        "protobuf",
+	DagCBOR:            "cbor",
+	GitRaw:             "git-raw",
+	EthBlock:           "eth-block",
+	EthBlockList:       "eth-block-list",
+	EthTxTrie:          "eth-tx-trie",
+	EthTx:              "eth-tx",
+	EthTxReceiptTrie:   "eth-tx-receipt-trie",
+	EthTxReceipt:       "eth-tx-receipt",
+	EthStateTrie:       "eth-state-trie",
+	EthAccountSnapshot: "eth-account-snapshot",
+	EthStorageTrie:     "eth-storage-trie",
+	BitcoinBlock:       "bitcoin-block",
+	BitcoinTx:          "bitcoin-tx",
+	ZcashBlock:         "zcash-block",
+	ZcashTx:            "zcash-tx",
+	DecredBlock:        "decred-block",
+	DecredTx:           "decred-tx",
+}

_rsrch/cidiface/errors.go

@@ -0,0 +1,24 @@
+package cid
+
+import (
+	"errors"
+)
+
+var (
+	// ErrVarintBuffSmall means that a buffer passed to the cid parser was not
+	// long enough, or did not contain an invalid cid
+	ErrVarintBuffSmall = errors.New("reading varint: buffer too small")
+
+	// ErrVarintTooBig means that the varint in the given cid was above the
+	// limit of 2^64
+	ErrVarintTooBig = errors.New("reading varint: varint bigger than 64bits" +
+		" and not supported")
+
+	// ErrCidTooShort means that the cid passed to decode was not long
+	// enough to be a valid Cid
+	ErrCidTooShort = errors.New("cid too short")
+
+	// ErrInvalidEncoding means that selected encoding is not supported
+	// by this Cid version
+	ErrInvalidEncoding = errors.New("invalid base encoding")
+)

_rsrch/cidiface/misc.go

@@ -0,0 +1,12 @@
+package cid
+
+func uvError(read int) error {
+	switch {
+	case read == 0:
+		return ErrVarintBuffSmall
+	case read < 0:
+		return ErrVarintTooBig
+	default:
+		return nil
+	}
+}

builder.go

@@ -0,0 +1,74 @@
+package cid
+
+import (
+	mh "github.com/multiformats/go-multihash"
+)
+
+type Builder interface {
+	Sum(data []byte) (Cid, error)
+	GetCodec() uint64
+	WithCodec(uint64) Builder
+}
+
+type V0Builder struct{}
+
+type V1Builder struct {
+	Codec    uint64
+	MhType   uint64
+	MhLength int // MhLength <= 0 means the default length
+}
+
+func (p Prefix) GetCodec() uint64 {
+	return p.Codec
+}
+
+func (p Prefix) WithCodec(c uint64) Builder {
+	if c == p.Codec {
+		return p
+	}
+	p.Codec = c
+	if c != DagProtobuf {
+		p.Version = 1
+	}
+	return p
+}
+
+func (p V0Builder) Sum(data []byte) (Cid, error) {
+	hash, err := mh.Sum(data, mh.SHA2_256, -1)
+	if err != nil {
+		return Undef, err
+	}
+	return NewCidV0(hash), nil
+}
+
+func (p V0Builder) GetCodec() uint64 {
+	return DagProtobuf
+}
+
+func (p V0Builder) WithCodec(c uint64) Builder {
+	if c == DagProtobuf {
+		return p
+	}
+	return V1Builder{Codec: c, MhType: mh.SHA2_256}
+}
+
+func (p V1Builder) Sum(data []byte) (Cid, error) {
+	mhLen := p.MhLength
+	if mhLen <= 0 {
+		mhLen = -1
+	}
+	hash, err := mh.Sum(data, p.MhType, mhLen)
+	if err != nil {
+		return Undef, err
+	}
+	return NewCidV1(p.Codec, hash), nil
+}
+
+func (p V1Builder) GetCodec() uint64 {
+	return p.Codec
+}
+
+func (p V1Builder) WithCodec(c uint64) Builder {
+	p.Codec = c
+	return p
+}

builder_test.go

@@ -0,0 +1,92 @@
+package cid
+
+import (
+	"testing"
+
+	mh "github.com/multiformats/go-multihash"
+)
+
+func TestV0Builder(t *testing.T) {
+	data := []byte("this is some test content")
+
+	// Construct c1
+	format := V0Builder{}
+	c1, err := format.Sum(data)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Construct c2
+	hash, err := mh.Sum(data, mh.SHA2_256, -1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	c2 := NewCidV0(hash)
+
+	if !c1.Equals(c2) {
+		t.Fatal("cids mismatch")
+	}
+	if c1.Prefix() != c2.Prefix() {
+		t.Fatal("prefixes mismatch")
+	}
+}
+
+func TestV1Builder(t *testing.T) {
+	data := []byte("this is some test content")
+
+	// Construct c1
+	format := V1Builder{Codec: DagCBOR, MhType: mh.SHA2_256}
+	c1, err := format.Sum(data)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Construct c2
+	hash, err := mh.Sum(data, mh.SHA2_256, -1)
+	if err != nil {
+		t.Fatal(err)
+	}
+	c2 := NewCidV1(DagCBOR, hash)
+
+	if !c1.Equals(c2) {
+		t.Fatal("cids mismatch")
+	}
+	if c1.Prefix() != c2.Prefix() {
+		t.Fatal("prefixes mismatch")
+	}
+}
+
+func TestCodecChange(t *testing.T) {
+	t.Run("Prefix-CidV0", func(t *testing.T) {
+		p := Prefix{Version: 0, Codec: DagProtobuf, MhType: mh.SHA2_256, MhLength: mh.DefaultLengths[mh.SHA2_256]}
+		testCodecChange(t, p)
+	})
+	t.Run("Prefix-CidV1", func(t *testing.T) {
+		p := Prefix{Version: 1, Codec: DagProtobuf, MhType: mh.SHA2_256, MhLength: mh.DefaultLengths[mh.SHA2_256]}
+		testCodecChange(t, p)
+	})
+	t.Run("V0Builder", func(t *testing.T) {
+		testCodecChange(t, V0Builder{})
+	})
+	t.Run("V1Builder", func(t *testing.T) {
+		testCodecChange(t, V1Builder{Codec: DagProtobuf, MhType: mh.SHA2_256})
+	})
+}
+
+func testCodecChange(t *testing.T, b Builder) {
+	data := []byte("this is some test content")
+
+	if b.GetCodec() != DagProtobuf {
+		t.Fatal("original builder not using Protobuf codec")
+	}
+
+	b = b.WithCodec(Raw)
+	c, err := b.Sum(data)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if c.Type() != Raw {
+		t.Fatal("new cid codec did not change to Raw")
+	}
+}

cid-fmt/main.go

@@ -1,277 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"fmt"
-	"os"
-	"strings"
-
-	c "github.com/ipfs/go-cid"
-
-	mb "github.com/multiformats/go-multibase"
-	mh "github.com/multiformats/go-multihash"
-)
-
-func usage() {
-	fmt.Fprintf(os.Stderr, "usage: %s [-b multibase-code] [-v cid-version] <fmt-str> <cid> ...\n\n", os.Args[0])
-	fmt.Fprintf(os.Stderr, "<fmt-str> is either 'prefix' or a printf style format string:\n%s", fmtRef)
-	os.Exit(2)
-}
-
-const fmtRef = `
-   %% literal %
-   %b multibase name 
-   %B multibase code
-   %v version string
-   %V version number
-   %c codec name
-   %C codec code
-   %h multihash name
-   %H multihash code
-   %L hash digest length
-   %m multihash encoded in base %b (with multibase prefix)
-   %M multihash encoded in base %b without multibase prefix
-   %d hash digest encoded in base %b (with multibase prefix)
-   %D hash digest encoded in base %b without multibase prefix
-   %s cid string encoded in base %b (1)
-   %s cid string encoded in base %b without multibase prefix
-   %P cid prefix: %v-%c-%h-%L
-
-(1) For CID version 0 the multibase must be base58btc and no prefix is
-used.  For Cid version 1 the multibase prefix is included.
-`
-
-func main() {
-	if len(os.Args) < 2 {
-		usage()
-	}
-	newBase := mb.Encoding(-1)
-	var verConv func(cid *c.Cid) (*c.Cid, error)
-	args := os.Args[1:]
-outer:
-	for {
-		switch args[0] {
-		case "-b":
-			if len(args) < 2 {
-				usage()
-			}
-			if len(args[1]) != 1 {
-				fmt.Fprintf(os.Stderr, "Error: Invalid multibase code: %s\n", args[1])
-				os.Exit(2)
-			}
-			newBase = mb.Encoding(args[1][0])
-			args = args[2:]
-		case "-v":
-			if len(args) < 2 {
-				usage()
-			}
-			switch args[1] {
-			case "0":
-				verConv = toCidV0
-			case "1":
-				verConv = toCidV1
-			default:
-				fmt.Fprintf(os.Stderr, "Error: Invalid cid version: %s\n", args[1])
-				os.Exit(2)
-			}
-			args = args[2:]
-		default:
-			break outer
-		}
-	}
-	if len(args) < 2 {
-		usage()
-	}
-	fmtStr := args[0]
-	switch fmtStr {
-	case "prefix":
-		fmtStr = "%P"
-	default:
-		if strings.IndexByte(fmtStr, '%') == -1 {
-			fmt.Fprintf(os.Stderr, "Error: Invalid format string: %s\n", fmtStr)
-			os.Exit(2)
-		}
-	}
-	for _, cidStr := range args[1:] {
-		base, cid, err := decode(cidStr)
-		if err != nil {
-			fmt.Fprintf(os.Stdout, "!INVALID_CID!\n")
-			errorMsg("%s: %v", cidStr, err)
-			// Don't abort on a bad cid
-			continue
-		}
-		if newBase != -1 {
-			base = newBase
-		}
-		if verConv != nil {
-			cid, err = verConv(cid)
-			if err != nil {
-				fmt.Fprintf(os.Stdout, "!ERROR!\n")
-				errorMsg("%s: %v", cidStr, err)
-				// Don't abort on a bad conversion
-				continue
-			}
-		}
-		str, err := fmtCid(fmtStr, base, cid)
-		if err != nil {
-			fmt.Fprintf(os.Stderr, "Error: %v\n", err)
-			// An error here means a bad format string, no point in continuing
-			os.Exit(2)
-		}
-		fmt.Fprintf(os.Stdout, "%s\n", str)
-	}
-	os.Exit(exitCode)
-}
-
-var exitCode = 0
-
-func errorMsg(fmtStr string, a ...interface{}) {
-	fmt.Fprintf(os.Stderr, "Error: ")
-	fmt.Fprintf(os.Stderr, fmtStr, a...)
-	fmt.Fprintf(os.Stderr, "\n")
-	exitCode = 1
-}
-
-func decode(v string) (mb.Encoding, *c.Cid, error) {
-	if len(v) < 2 {
-		return 0, nil, c.ErrCidTooShort
-	}
-
-	if len(v) == 46 && v[:2] == "Qm" {
-		hash, err := mh.FromB58String(v)
-		if err != nil {
-			return 0, nil, err
-		}
-
-		return mb.Base58BTC, c.NewCidV0(hash), nil
-	}
-
-	base, data, err := mb.Decode(v)
-	if err != nil {
-		return 0, nil, err
-	}
-
-	cid, err := c.Cast(data)
-
-	return base, cid, err
-}
-
-const ERR_STR = "!ERROR!"
-
-func fmtCid(fmtStr string, base mb.Encoding, cid *c.Cid) (string, error) {
-	p := cid.Prefix()
-	out := new(bytes.Buffer)
-	var err error
-	for i := 0; i < len(fmtStr); i++ {
-		if fmtStr[i] != '%' {
-			out.WriteByte(fmtStr[i])
-			continue
-		}
-		i++
-		if i >= len(fmtStr) {
-			return "", fmt.Errorf("premature end of format string")
-		}
-		switch fmtStr[i] {
-		case '%':
-			out.WriteByte('%')
-		case 'b': // base name
-			out.WriteString(baseToString(base))
-		case 'B': // base code
-			out.WriteByte(byte(base))
-		case 'v': // version string
-			fmt.Fprintf(out, "cidv%d", p.Version)
-		case 'V': // version num
-			fmt.Fprintf(out, "%d", p.Version)
-		case 'c': // codec name
-			out.WriteString(codecToString(p.Codec))
-		case 'C': // codec code
-			fmt.Fprintf(out, "%d", p.Codec)
-		case 'h': // hash fun name
-			out.WriteString(hashToString(p.MhType))
-		case 'H': // hash fun code
-			fmt.Fprintf(out, "%d", p.MhType)
-		case 'L': // hash length
-			fmt.Fprintf(out, "%d", p.MhLength)
-		case 'm', 'M': // multihash encoded in base %b
-			out.WriteString(encode(base, cid.Hash(), fmtStr[i] == 'M'))
-		case 'd', 'D': // hash digest encoded in base %b
-			dec, err := mh.Decode(cid.Hash())
-			if err != nil {
-				out.WriteString(ERR_STR)
-				errorMsg("%v", err)
-				continue
-			}
-			out.WriteString(encode(base, dec.Digest, fmtStr[i] == 'D'))
-		case 's': // cid string encoded in base %b
-			str, err := cid.StringOfBase(base)
-			if err != nil {
-				out.WriteString(ERR_STR)
-				errorMsg("%v", err)
-				continue
-			}
-			out.WriteString(str)
-		case 'S': // cid string without base prefix
-			out.WriteString(encode(base, cid.Bytes(), true))
-		case 'P': // prefix
-			fmt.Fprintf(out, "cidv%d-%s-%s-%d",
-				p.Version,
-				codecToString(p.Codec),
-				hashToString(p.MhType),
-				p.MhLength,
-			)
-		default:
-			return "", fmt.Errorf("unrecognized specifier in format string: %c", fmtStr[i])
-		}
-
-	}
-	return out.String(), err
-}
-
-func baseToString(base mb.Encoding) string {
-	// FIXME: Use lookup tables when they are added to go-multibase
-	switch base {
-	case mb.Base58BTC:
-		return "base58btc"
-	default:
-		return fmt.Sprintf("base?%c", base)
-	}
-}
-
-func codecToString(num uint64) string {
-	name, ok := c.CodecToStr[num]
-	if !ok {
-		return fmt.Sprintf("codec?%d", num)
-	}
-	return name
-}
-
-func hashToString(num uint64) string {
-	name, ok := mh.Codes[num]
-	if !ok {
-		return fmt.Sprintf("hash?%d", num)
-	}
-	return name
-}
-
-func encode(base mb.Encoding, data []byte, strip bool) string {
-	str, err := mb.Encode(base, data)
-	if err != nil {
-		errorMsg("%v", err)
-		return ERR_STR
-	}
-	if strip {
-		return str[1:]
-	}
-	return str
-}
-
-func toCidV0(cid *c.Cid) (*c.Cid, error) {
-	if cid.Type() != c.DagProtobuf {
-		return nil, fmt.Errorf("can't convert non-protobuf nodes to cidv0")
-	}
-	return c.NewCidV0(cid.Hash()), nil
-}
-
-func toCidV1(cid *c.Cid) (*c.Cid, error) {
-	return c.NewCidV1(cid.Type(), cid.Hash()), nil
-}

cid-fmt/main_test.go

@@ -1,109 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"testing"
-
-	mb "github.com/multiformats/go-multibase"
-)
-
-func TestFmt(t *testing.T) {
-	cids := map[string]string{
-		"cidv0": "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn",
-		"cidv1": "zdj7WfLr9DhLrb1hsoSi4fSdjjxuZmeqgEtBPWxMLtPbDNbFD",
-	}
-	tests := []struct {
-		cidId   string
-		newBase mb.Encoding
-		fmtStr  string
-		result  string
-	}{
-		{"cidv0", -1, "%P", "cidv0-protobuf-sha2-256-32"},
-		{"cidv0", -1, "%b-%v-%c-%h-%L", "base58btc-cidv0-protobuf-sha2-256-32"},
-		{"cidv0", -1, "%s", "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"},
-		{"cidv0", -1, "%S", "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"},
-		{"cidv0", -1, "ver#%V/#%C/#%H/%L", "ver#0/#112/#18/32"},
-		{"cidv0", -1, "%m", "zQmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"},
-		{"cidv0", -1, "%M", "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"},
-		{"cidv0", -1, "%d", "z72gdmFAgRzYHkJzKiL8MgMMRW3BTSCGyDHroPxJbxMJn"},
-		{"cidv0", -1, "%D", "72gdmFAgRzYHkJzKiL8MgMMRW3BTSCGyDHroPxJbxMJn"},
-		{"cidv0", 'B', "%S", "CIQFTFEEHEDF6KLBT32BFAGLXEZL4UWFNWM4LFTLMXQBCERZ6CMLX3Y"},
-		{"cidv0", 'B', "%B%S", "BCIQFTFEEHEDF6KLBT32BFAGLXEZL4UWFNWM4LFTLMXQBCERZ6CMLX3Y"},
-		{"cidv1", -1, "%P", "cidv1-protobuf-sha2-256-32"},
-		{"cidv1", -1, "%b-%v-%c-%h-%L", "base58btc-cidv1-protobuf-sha2-256-32"},
-		{"cidv1", -1, "%s", "zdj7WfLr9DhLrb1hsoSi4fSdjjxuZmeqgEtBPWxMLtPbDNbFD"},
-		{"cidv1", -1, "%S", "dj7WfLr9DhLrb1hsoSi4fSdjjxuZmeqgEtBPWxMLtPbDNbFD"},
-		{"cidv1", -1, "ver#%V/#%C/#%H/%L", "ver#1/#112/#18/32"},
-		{"cidv1", -1, "%m", "zQmYFbmndVP7QqAVWyKhpmMuQHMaD88pkK57RgYVimmoh5H"},
-		{"cidv1", -1, "%M", "QmYFbmndVP7QqAVWyKhpmMuQHMaD88pkK57RgYVimmoh5H"},
-		{"cidv1", -1, "%d", "zAux4gVVsLRMXtsZ9fd3tFEZN4jGYB6kP37fgoZNTc11H"},
-		{"cidv1", -1, "%D", "Aux4gVVsLRMXtsZ9fd3tFEZN4jGYB6kP37fgoZNTc11H"},
-		{"cidv1", 'B', "%s", "BAFYBEIETJGSRL3EQPQPCABV3G6IUBYTSIFVQ24XRRHD3JUETSKLTGQ7DJA"},
-		{"cidv1", 'B', "%S", "AFYBEIETJGSRL3EQPQPCABV3G6IUBYTSIFVQ24XRRHD3JUETSKLTGQ7DJA"},
-		{"cidv1", 'B', "%B%S", "BAFYBEIETJGSRL3EQPQPCABV3G6IUBYTSIFVQ24XRRHD3JUETSKLTGQ7DJA"},
-	}
-	for _, tc := range tests {
-		name := fmt.Sprintf("%s/%s", tc.cidId, tc.fmtStr)
-		if tc.newBase != -1 {
-			name = fmt.Sprintf("%s/%c", name, tc.newBase)
-		}
-		cidStr := cids[tc.cidId]
-		t.Run(name, func(t *testing.T) {
-			testFmt(t, cidStr, tc.newBase, tc.fmtStr, tc.result)
-		})
-	}
-}
-
-func testFmt(t *testing.T, cidStr string, newBase mb.Encoding, fmtStr string, result string) {
-	base, cid, err := decode(cidStr)
-	if newBase != -1 {
-		base = newBase
-	}
-	if err != nil {
-		t.Fatal(err)
-	}
-	str, err := fmtCid(fmtStr, base, cid)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if str != result {
-		t.Error(fmt.Sprintf("expected: %s; but got: %s", result, str))
-	}
-}
-
-func TestCidConv(t *testing.T) {
-	cidv0 := "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"
-	cidv1 := "zdj7WbTaiJT1fgatdet9Ei9iDB5hdCxkbVyhyh8YTUnXMiwYi"
-	_, cid, err := decode(cidv0)
-	if err != nil {
-		t.Fatal(err)
-	}
-	cid, err = toCidV1(cid)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if cid.String() != cidv1 {
-		t.Fatal("conversion failure")
-	}
-	cid, err = toCidV0(cid)
-	if err != nil {
-		t.Fatal(err)
-	}
-	cidStr := cid.String()
-	if cidStr != cidv0 {
-		t.Error(fmt.Sprintf("conversion failure, expected: %s; but got: %s", cidv0, cidStr))
-	}
-}
-
-func TestBadCidConv(t *testing.T) {
-	// this cid is a raw leaf and should not be able to convert to cidv0
-	cidv1 := "zb2rhhzX7uSKrtQ2ZZXFAabKiKFYZrJqKY2KE1cJ8yre2GSWZ"
-	_, cid, err := decode(cidv1)
-	if err != nil {
-		t.Fatal(err)
-	}
-	cid, err = toCidV0(cid)
-	if err == nil {
-		t.Fatal("expected failure")
-	}
-}

cid.go

@@ -77,6 +77,8 @@ const (
 	BitcoinTx          = 0xb1
 	ZcashBlock         = 0xc0
 	ZcashTx            = 0xc1
+	DecredBlock        = 0xe0
+	DecredTx           = 0xe1
 )
 
 // Codecs maps the name of a codec to its type
@@ -99,6 +101,8 @@ var Codecs = map[string]uint64{
 	"bitcoin-tx":           BitcoinTx,
 	"zcash-block":          ZcashBlock,
 	"zcash-tx":             ZcashTx,
+	"decred-block":         DecredBlock,
+	"decred-tx":            DecredTx,
 }
 
 // CodecToStr maps the numeric codec to its name
@@ -120,63 +124,62 @@ var CodecToStr = map[uint64]string{
 	BitcoinTx:          "bitcoin-tx",
 	ZcashBlock:         "zcash-block",
 	ZcashTx:            "zcash-tx",
+	DecredBlock:        "decred-block",
+	DecredTx:           "decred-tx",
 }
 
 // NewCidV0 returns a Cid-wrapped multihash.
 // They exist to allow IPFS to work with Cids while keeping
 // compatibility with the plain-multihash format used used in IPFS.
 // NewCidV1 should be used preferentially.
-func NewCidV0(mhash mh.Multihash) *Cid {
+func NewCidV0(mhash mh.Multihash) Cid {
-	return &Cid{
+	// Need to make sure hash is valid for CidV0 otherwise we will
-		version: 0,
+	// incorrectly detect it as CidV1 in the Version() method
-		codec:   DagProtobuf,
+	dec, err := mh.Decode(mhash)
-		hash:    mhash,
+	if err != nil {
+		panic(err)
 	}
+	if dec.Code != mh.SHA2_256 || dec.Length != 32 {
+		panic("invalid hash for cidv0")
+	}
+	return Cid{string(mhash)}
 }
 
 // NewCidV1 returns a new Cid using the given multicodec-packed
 // content type.
-func NewCidV1(codecType uint64, mhash mh.Multihash) *Cid {
+func NewCidV1(codecType uint64, mhash mh.Multihash) Cid {
-	return &Cid{
+	hashlen := len(mhash)
-		version: 1,
+	// two 8 bytes (max) numbers plus hash
-		codec:   codecType,
+	buf := make([]byte, 2*binary.MaxVarintLen64+hashlen)
-		hash:    mhash,
+	n := binary.PutUvarint(buf, 1)
+	n += binary.PutUvarint(buf[n:], codecType)
+	cn := copy(buf[n:], mhash)
+	if cn != hashlen {
+		panic("copy hash length is inconsistent")
 	}
-}
 
-// NewPrefixV0 returns a CIDv0 prefix with the specified multihash type.
+	return Cid{string(buf[:n+hashlen])}
-func NewPrefixV0(mhType uint64) Prefix {
-	return Prefix{
-		MhType:   mhType,
-		MhLength: mh.DefaultLengths[mhType],
-		Version:  0,
-		Codec:    DagProtobuf,
-	}
-}
-
-// NewPrefixV1 returns a CIDv1 prefix with the specified codec and multihash
-// type.
-func NewPrefixV1(codecType uint64, mhType uint64) Prefix {
-	return Prefix{
-		MhType:   mhType,
-		MhLength: mh.DefaultLengths[mhType],
-		Version:  1,
-		Codec:    codecType,
-	}
 }
 
 // Cid represents a self-describing content adressed
 // identifier. It is formed by a Version, a Codec (which indicates
 // a multicodec-packed content type) and a Multihash.
-type Cid struct {
+type Cid struct{ str string }
-	version uint64
+
-	codec   uint64
+// Undef can be used to represent a nil or undefined Cid, using Cid{}
-	hash    mh.Multihash
+// directly is also acceptable.
+var Undef = Cid{}
+
+// Defined returns true if a Cid is defined
+// Calling any other methods on an undefined Cid will result in
+// undefined behavior.
+func (c Cid) Defined() bool {
+	return c.str != ""
 }
 
 // Parse is a short-hand function to perform Decode, Cast etc... on
 // a generic interface{} type.
-func Parse(v interface{}) (*Cid, error) {
+func Parse(v interface{}) (Cid, error) {
 	switch v2 := v.(type) {
 	case string:
 		if strings.Contains(v2, "/ipfs/") {
@@ -187,10 +190,10 @@ func Parse(v interface{}) (*Cid, error) {
 		return Cast(v2)
 	case mh.Multihash:
 		return NewCidV0(v2), nil
-	case *Cid:
+	case Cid:
 		return v2, nil
 	default:
-		return nil, fmt.Errorf("can't parse %+v as Cid", v2)
+		return Undef, fmt.Errorf("can't parse %+v as Cid", v2)
 	}
 }
 
@@ -206,15 +209,15 @@ func Parse(v interface{}) (*Cid, error) {
 // Decode will also detect and parse CidV0 strings. Strings
 // starting with "Qm" are considered CidV0 and treated directly
 // as B58-encoded multihashes.
-func Decode(v string) (*Cid, error) {
+func Decode(v string) (Cid, error) {
 	if len(v) < 2 {
-		return nil, ErrCidTooShort
+		return Undef, ErrCidTooShort
 	}
 
 	if len(v) == 46 && v[:2] == "Qm" {
 		hash, err := mh.FromB58String(v)
 		if err != nil {
-			return nil, err
+			return Undef, err
 		}
 
 		return NewCidV0(hash), nil
@@ -222,12 +225,34 @@ func Decode(v string) (*Cid, error) {
 
 	_, data, err := mbase.Decode(v)
 	if err != nil {
-		return nil, err
+		return Undef, err
 	}
 
 	return Cast(data)
 }
 
+// Extract the encoding from a Cid.  If Decode on the same string did
+// not return an error neither will this function.
+func ExtractEncoding(v string) (mbase.Encoding, error) {
+	if len(v) < 2 {
+		return -1, ErrCidTooShort
+	}
+
+	if len(v) == 46 && v[:2] == "Qm" {
+		return mbase.Base58BTC, nil
+	}
+
+	encoding := mbase.Encoding(v[0])
+
+	// check encoding is valid
+	_, err := mbase.NewEncoder(encoding)
+	if err != nil {
+		return -1, err
+	}
+
+	return encoding, nil
+}
+
 func uvError(read int) error {
 	switch {
 	case read == 0:
@@ -250,61 +275,66 @@ func uvError(read int) error {
 //
 // Please use decode when parsing a regular Cid string, as Cast does not
 // expect multibase-encoded data. Cast accepts the output of Cid.Bytes().
-func Cast(data []byte) (*Cid, error) {
+func Cast(data []byte) (Cid, error) {
 	if len(data) == 34 && data[0] == 18 && data[1] == 32 {
 		h, err := mh.Cast(data)
 		if err != nil {
-			return nil, err
+			return Undef, err
 		}
 
-		return &Cid{
+		return NewCidV0(h), nil
-			codec:   DagProtobuf,
-			version: 0,
-			hash:    h,
-		}, nil
 	}
 
 	vers, n := binary.Uvarint(data)
 	if err := uvError(n); err != nil {
-		return nil, err
+		return Undef, err
 	}
 
-	if vers != 0 && vers != 1 {
+	if vers != 1 {
-		return nil, fmt.Errorf("invalid cid version number: %d", vers)
+		return Undef, fmt.Errorf("expected 1 as the cid version number, got: %d", vers)
 	}
 
-	codec, cn := binary.Uvarint(data[n:])
+	_, cn := binary.Uvarint(data[n:])
 	if err := uvError(cn); err != nil {
-		return nil, err
+		return Undef, err
 	}
 
 	rest := data[n+cn:]
 	h, err := mh.Cast(rest)
 	if err != nil {
-		return nil, err
+		return Undef, err
 	}
 
-	return &Cid{
+	return Cid{string(data[0 : n+cn+len(h)])}, nil
-		version: vers,
+}
-		codec:   codec,
+
-		hash:    h,
+// Version returns the Cid version.
-	}, nil
+func (c Cid) Version() uint64 {
+	if len(c.str) == 34 && c.str[0] == 18 && c.str[1] == 32 {
+		return 0
+	}
+	return 1
 }
 
 // Type returns the multicodec-packed content type of a Cid.
-func (c *Cid) Type() uint64 {
+func (c Cid) Type() uint64 {
-	return c.codec
+	if c.Version() == 0 {
+		return DagProtobuf
+	}
+	_, n := uvarint(c.str)
+	codec, _ := uvarint(c.str[n:])
+	return codec
 }
 
 // String returns the default string representation of a
 // Cid. Currently, Base58 is used as the encoding for the
 // multibase string.
-func (c *Cid) String() string {
+func (c Cid) String() string {
-	switch c.version {
+	switch c.Version() {
 	case 0:
-		return c.hash.B58String()
+		return c.Hash().B58String()
 	case 1:
-		mbstr, err := mbase.Encode(mbase.Base58BTC, c.bytesV1())
+		mbstr, err := mbase.Encode(mbase.Base58BTC, c.Bytes())
 		if err != nil {
 			panic("should not error with hardcoded mbase: " + err.Error())
 		}
@@ -317,63 +347,62 @@ func (c *Cid) String() string {
 
 // String returns the string representation of a Cid
 // encoded is selected base
-func (c *Cid) StringOfBase(base mbase.Encoding) (string, error) {
+func (c Cid) StringOfBase(base mbase.Encoding) (string, error) {
-	switch c.version {
+	switch c.Version() {
 	case 0:
 		if base != mbase.Base58BTC {
 			return "", ErrInvalidEncoding
 		}
-		return c.hash.B58String(), nil
+		return c.Hash().B58String(), nil
 	case 1:
-		return mbase.Encode(base, c.bytesV1())
+		return mbase.Encode(base, c.Bytes())
+	default:
+		panic("not possible to reach this point")
+	}
+}
+
+// Encode return the string representation of a Cid in a given base
+// when applicable.  Version 0 Cid's are always in Base58 as they do
+// not take a multibase prefix.
+func (c Cid) Encode(base mbase.Encoder) string {
+	switch c.Version() {
+	case 0:
+		return c.Hash().B58String()
+	case 1:
+		return base.Encode(c.Bytes())
 	default:
 		panic("not possible to reach this point")
 	}
 }
 
 // Hash returns the multihash contained by a Cid.
-func (c *Cid) Hash() mh.Multihash {
+func (c Cid) Hash() mh.Multihash {
-	return c.hash
+	bytes := c.Bytes()
+
+	if c.Version() == 0 {
+		return mh.Multihash(bytes)
+	}
+
+	// skip version length
+	_, n1 := binary.Uvarint(bytes)
+	// skip codec length
+	_, n2 := binary.Uvarint(bytes[n1:])
+
+	return mh.Multihash(bytes[n1+n2:])
 }
 
 // Bytes returns the byte representation of a Cid.
 // The output of bytes can be parsed back into a Cid
 // with Cast().
-func (c *Cid) Bytes() []byte {
+func (c Cid) Bytes() []byte {
-	switch c.version {
+	return []byte(c.str)
-	case 0:
-		return c.bytesV0()
-	case 1:
-		return c.bytesV1()
-	default:
-		panic("not possible to reach this point")
-	}
-}
-
-func (c *Cid) bytesV0() []byte {
-	return []byte(c.hash)
-}
-
-func (c *Cid) bytesV1() []byte {
-	// two 8 bytes (max) numbers plus hash
-	buf := make([]byte, 2*binary.MaxVarintLen64+len(c.hash))
-	n := binary.PutUvarint(buf, c.version)
-	n += binary.PutUvarint(buf[n:], c.codec)
-	cn := copy(buf[n:], c.hash)
-	if cn != len(c.hash) {
-		panic("copy hash length is inconsistent")
-	}
-
-	return buf[:n+len(c.hash)]
 }
 
 // Equals checks that two Cids are the same.
 // In order for two Cids to be considered equal, the
 // Version, the Codec and the Multihash must match.
-func (c *Cid) Equals(o *Cid) bool {
+func (c Cid) Equals(o Cid) bool {
-	return c.codec == o.codec &&
+	return c == o
-		c.version == o.version &&
-		bytes.Equal(c.hash, o.hash)
 }
 
 // UnmarshalJSON parses the JSON representation of a Cid.
@@ -384,10 +413,15 @@ func (c *Cid) UnmarshalJSON(b []byte) error {
 	obj := struct {
 		CidTarget string `json:"/"`
 	}{}
-	err := json.Unmarshal(b, &obj)
+	objptr := &obj
+	err := json.Unmarshal(b, &objptr)
 	if err != nil {
 		return err
 	}
+	if objptr == nil {
+		*c = Cid{}
+		return nil
+	}
 
 	if obj.CidTarget == "" {
 		return fmt.Errorf("cid was incorrectly formatted")
@@ -398,9 +432,8 @@ func (c *Cid) UnmarshalJSON(b []byte) error {
 		return err
 	}
 
-	c.version = out.version
+	*c = out
-	c.hash = out.hash
+
-	c.codec = out.codec
 	return nil
 }
 
@@ -410,31 +443,34 @@ func (c *Cid) UnmarshalJSON(b []byte) error {
 //
 // Note that this formatting comes from the IPLD specification
 // (https://github.com/ipld/specs/tree/master/ipld)
-func (c *Cid) MarshalJSON() ([]byte, error) {
+func (c Cid) MarshalJSON() ([]byte, error) {
+	if !c.Defined() {
+		return []byte("null"), nil
+	}
 	return []byte(fmt.Sprintf("{\"/\":\"%s\"}", c.String())), nil
 }
 
-// KeyString casts the result of cid.Bytes() as a string, and returns it.
+// KeyString returns the binary representation of the Cid as a string
-func (c *Cid) KeyString() string {
+func (c Cid) KeyString() string {
-	return string(c.Bytes())
+	return c.str
 }
 
 // Loggable returns a Loggable (as defined by
 // https://godoc.org/github.com/ipfs/go-log).
-func (c *Cid) Loggable() map[string]interface{} {
+func (c Cid) Loggable() map[string]interface{} {
 	return map[string]interface{}{
 		"cid": c,
 	}
 }
 
 // Prefix builds and returns a Prefix out of a Cid.
-func (c *Cid) Prefix() Prefix {
+func (c Cid) Prefix() Prefix {
-	dec, _ := mh.Decode(c.hash) // assuming we got a valid multiaddr, this will not error
+	dec, _ := mh.Decode(c.Hash()) // assuming we got a valid multiaddr, this will not error
 	return Prefix{
 		MhType:   dec.Code,
 		MhLength: dec.Length,
-		Version:  c.version,
+		Version:  c.Version(),
-		Codec:    c.codec,
+		Codec:    c.Type(),
 	}
 }
 
@@ -442,6 +478,8 @@ func (c *Cid) Prefix() Prefix {
 // that is, the Version, the Codec, the Multihash type
 // and the Multihash length. It does not contains
 // any actual content information.
+// NOTE: The use -1 in MhLength to mean default length is deprecated,
+//   use the V0Builder or V1Builder structures instead
 type Prefix struct {
 	Version  uint64
 	Codec    uint64
@@ -451,10 +489,10 @@ type Prefix struct {
 
 // Sum uses the information in a prefix to perform a multihash.Sum()
 // and return a newly constructed Cid with the resulting multihash.
-func (p Prefix) Sum(data []byte) (*Cid, error) {
+func (p Prefix) Sum(data []byte) (Cid, error) {
 	hash, err := mh.Sum(data, p.MhType, p.MhLength)
 	if err != nil {
-		return nil, err
+		return Undef, err
 	}
 
 	switch p.Version {
@@ -463,7 +501,7 @@ func (p Prefix) Sum(data []byte) (*Cid, error) {
 	case 1:
 		return NewCidV1(p.Codec, hash), nil
 	default:
-		return nil, fmt.Errorf("invalid cid version")
+		return Undef, fmt.Errorf("invalid cid version")
 	}
 }
 

cid_fuzz.go

@@ -23,7 +23,7 @@ func Fuzz(data []byte) int {
 	if err != nil {
 		panic(err.Error())
 	}
-	cid2 := &Cid{}
+	cid2 := Cid{}
 	err = cid2.UnmarshalJSON(json)
 	if err != nil {
 		panic(err.Error())

cid_test.go

@@ -33,18 +33,20 @@ var tCodecs = map[uint64]string{
 	BitcoinTx:          "bitcoin-tx",
 	ZcashBlock:         "zcash-block",
 	ZcashTx:            "zcash-tx",
+	DecredBlock:        "decred-block",
+	DecredTx:           "decred-tx",
 }
 
-func assertEqual(t *testing.T, a, b *Cid) {
+func assertEqual(t *testing.T, a, b Cid) {
-	if a.codec != b.codec {
+	if a.Type() != b.Type() {
 		t.Fatal("mismatch on type")
 	}
 
-	if a.version != b.version {
+	if a.Version() != b.Version() {
 		t.Fatal("mismatch on version")
 	}
 
-	if !bytes.Equal(a.hash, b.hash) {
+	if !bytes.Equal(a.Hash(), b.Hash()) {
 		t.Fatal("multihash mismatch")
 	}
 }
@@ -75,11 +77,7 @@ func TestBasicMarshaling(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	cid := &Cid{
+	cid := NewCidV1(7, h)
-		codec:   7,
-		version: 1,
-		hash:    h,
-	}
 
 	data := cid.Bytes()
 
@@ -105,11 +103,7 @@ func TestBasesMarshaling(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	cid := &Cid{
+	cid := NewCidV1(7, h)
-		codec:   7,
-		version: 1,
-		hash:    h,
-	}
 
 	data := cid.Bytes()
 
@@ -150,6 +144,15 @@ func TestBasesMarshaling(t *testing.T) {
 		}
 
 		assertEqual(t, cid, out2)
+
+		encoder, err := mbase.NewEncoder(b)
+		if err != nil {
+			t.Fatal(err)
+		}
+		s2 := cid.Encode(encoder)
+		if s != s2 {
+			t.Fatalf("'%s' != '%s'", s, s2)
+		}
 	}
 }
 
@@ -168,17 +171,33 @@ func TestV0Handling(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	if cid.version != 0 {
+	if cid.Version() != 0 {
 		t.Fatal("should have gotten version 0 cid")
 	}
 
-	if cid.hash.B58String() != old {
+	if cid.Hash().B58String() != old {
-		t.Fatal("marshaling roundtrip failed")
+		t.Fatalf("marshaling roundtrip failed: %s != %s", cid.Hash().B58String(), old)
 	}
 
 	if cid.String() != old {
 		t.Fatal("marshaling roundtrip failed")
 	}
+
+	new, err := cid.StringOfBase(mbase.Base58BTC)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if new != old {
+		t.Fatal("StringOfBase roundtrip failed")
+	}
+
+	encoder, err := mbase.NewEncoder(mbase.Base58BTC)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if cid.Encode(encoder) != old {
+		t.Fatal("Encode roundtrip failed")
+	}
 }
 
 func TestV0ErrorCases(t *testing.T) {
@@ -279,9 +298,7 @@ func TestPrefixRoundtrip(t *testing.T) {
 func Test16BytesVarint(t *testing.T) {
 	data := []byte("this is some test content")
 	hash, _ := mh.Sum(data, mh.SHA2_256, -1)
-	c := NewCidV1(DagCBOR, hash)
+	c := NewCidV1(1<<63, hash)
-
-	c.codec = 1 << 63
 	_ = c.Bytes()
 }
 
@@ -324,8 +341,8 @@ func TestParse(t *testing.T) {
 		if err != nil {
 			return err
 		}
-		if cid.version != 0 {
+		if cid.Version() != 0 {
-			return fmt.Errorf("expected version 0, got %s", string(cid.version))
+			return fmt.Errorf("expected version 0, got %s", string(cid.Version()))
 		}
 		actual := cid.Hash().B58String()
 		if actual != expected {
@@ -383,3 +400,49 @@ func TestFromJson(t *testing.T) {
 		t.Fatal("json parsing failed")
 	}
 }
+
+func TestJsonRoundTrip(t *testing.T) {
+	exp, err := Decode("zb2rhhFAEMepUBbGyP1k8tGfz7BSciKXP6GHuUeUsJBaK6cqG")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Verify it works for a *Cid.
+	enc, err := json.Marshal(exp)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var actual Cid
+	err = json.Unmarshal(enc, &actual)
+	if !exp.Equals(actual) {
+		t.Fatal("cids not equal for *Cid")
+	}
+
+	// Verify it works for a Cid.
+	enc, err = json.Marshal(exp)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var actual2 Cid
+	err = json.Unmarshal(enc, &actual2)
+	if !exp.Equals(actual2) {
+		t.Fatal("cids not equal for Cid")
+	}
+}
+
+func BenchmarkStringV1(b *testing.B) {
+	data := []byte("this is some test content")
+	hash, _ := mh.Sum(data, mh.SHA2_256, -1)
+	cid := NewCidV1(Raw, hash)
+
+	b.ReportAllocs()
+	b.ResetTimer()
+
+	count := 0
+	for i := 0; i < b.N; i++ {
+		count += len(cid.String())
+	}
+	if count != 49*b.N {
+		b.FailNow()
+	}
+}

deprecated.go

@@ -0,0 +1,28 @@
+package cid
+
+import (
+	mh "github.com/multiformats/go-multihash"
+)
+
+// NewPrefixV0 returns a CIDv0 prefix with the specified multihash type.
+// DEPRECATED: Use V0Builder
+func NewPrefixV0(mhType uint64) Prefix {
+	return Prefix{
+		MhType:   mhType,
+		MhLength: mh.DefaultLengths[mhType],
+		Version:  0,
+		Codec:    DagProtobuf,
+	}
+}
+
+// NewPrefixV1 returns a CIDv1 prefix with the specified codec and multihash
+// type.
+// DEPRECATED: Use V1Builder
+func NewPrefixV1(codecType uint64, mhType uint64) Prefix {
+	return Prefix{
+		MhType:   mhType,
+		MhLength: mh.DefaultLengths[mhType],
+		Version:  1,
+		Codec:    codecType,
+	}
+}

package.json

@@ -9,15 +9,15 @@
   "gxDependencies": [
     {
       "author": "whyrusleeping",
-      "hash": "QmYeKnKpubCMRiq3PGZcTREErthbb5Q9cXsCoSkD9bjEBd",
+      "hash": "QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8",
       "name": "go-multihash",
-      "version": "1.0.6"
+      "version": "1.0.8"
     },
     {
       "author": "whyrusleeping",
-      "hash": "QmafgXF3u3QSWErQoZ2URmQp5PFG384htoE7J338nS2H7T",
+      "hash": "QmekxXDhCxCJRNuzmHreuaT3BsuJcsjcXWNrtV9C8DRHtd",
       "name": "go-multibase",
-      "version": "0.2.5"
+      "version": "0.3.0"
     }
   ],
   "gxVersion": "0.8.0",
@@ -25,6 +25,6 @@
   "license": "MIT",
   "name": "go-cid",
   "releaseCmd": "git commit -a -m \"gx publish $VERSION\"",
-  "version": "0.7.19"
+  "version": "0.9.0"
 }
 

set.go

@@ -3,28 +3,28 @@ package cid
 // Set is a implementation of a set of Cids, that is, a structure
 // to which holds a single copy of every Cids that is added to it.
 type Set struct {
-	set map[string]struct{}
+	set map[Cid]struct{}
 }
 
 // NewSet initializes and returns a new Set.
 func NewSet() *Set {
-	return &Set{set: make(map[string]struct{})}
+	return &Set{set: make(map[Cid]struct{})}
 }
 
 // Add puts a Cid in the Set.
-func (s *Set) Add(c *Cid) {
+func (s *Set) Add(c Cid) {
-	s.set[string(c.Bytes())] = struct{}{}
+	s.set[c] = struct{}{}
 }
 
 // Has returns if the Set contains a given Cid.
-func (s *Set) Has(c *Cid) bool {
+func (s *Set) Has(c Cid) bool {
-	_, ok := s.set[string(c.Bytes())]
+	_, ok := s.set[c]
 	return ok
 }
 
 // Remove deletes a Cid from the Set.
-func (s *Set) Remove(c *Cid) {
+func (s *Set) Remove(c Cid) {
-	delete(s.set, string(c.Bytes()))
+	delete(s.set, c)
 }
 
 // Len returns how many elements the Set has.
@@ -33,18 +33,17 @@ func (s *Set) Len() int {
 }
 
 // Keys returns the Cids in the set.
-func (s *Set) Keys() []*Cid {
+func (s *Set) Keys() []Cid {
-	out := make([]*Cid, 0, len(s.set))
+	out := make([]Cid, 0, len(s.set))
 	for k := range s.set {
-		c, _ := Cast([]byte(k))
+		out = append(out, k)
-		out = append(out, c)
 	}
 	return out
 }
 
 // Visit adds a Cid to the set only if it is
 // not in it already.
-func (s *Set) Visit(c *Cid) bool {
+func (s *Set) Visit(c Cid) bool {
 	if !s.Has(c) {
 		s.Add(c)
 		return true
@@ -55,9 +54,8 @@ func (s *Set) Visit(c *Cid) bool {
 
 // ForEach allows to run a custom function on each
 // Cid in the set.
-func (s *Set) ForEach(f func(c *Cid) error) error {
+func (s *Set) ForEach(f func(c Cid) error) error {
-	for cs := range s.set {
+	for c := range s.set {
-		c, _ := Cast([]byte(cs))
 		err := f(c)
 		if err != nil {
 			return err

set_test.go

@@ -0,0 +1,88 @@
+package cid
+
+import (
+	"crypto/rand"
+	"errors"
+	"testing"
+
+	mh "github.com/multiformats/go-multihash"
+)
+
+func makeRandomCid(t *testing.T) Cid {
+	p := make([]byte, 256)
+	_, err := rand.Read(p)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	h, err := mh.Sum(p, mh.SHA3, 4)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	cid := NewCidV1(7, h)
+
+	return cid
+}
+
+func TestSet(t *testing.T) {
+	cid := makeRandomCid(t)
+	cid2 := makeRandomCid(t)
+	s := NewSet()
+
+	s.Add(cid)
+
+	if !s.Has(cid) {
+		t.Error("should have the CID")
+	}
+
+	if s.Len() != 1 {
+		t.Error("should report 1 element")
+	}
+
+	keys := s.Keys()
+
+	if len(keys) != 1 || !keys[0].Equals(cid) {
+		t.Error("key should correspond to Cid")
+	}
+
+	if s.Visit(cid) {
+		t.Error("visit should return false")
+	}
+
+	foreach := []Cid{}
+	foreachF := func(c Cid) error {
+		foreach = append(foreach, c)
+		return nil
+	}
+
+	if err := s.ForEach(foreachF); err != nil {
+		t.Error(err)
+	}
+
+	if len(foreach) != 1 {
+		t.Error("ForEach should have visited 1 element")
+	}
+
+	foreachErr := func(c Cid) error {
+		return errors.New("test")
+	}
+
+	if err := s.ForEach(foreachErr); err == nil {
+		t.Error("Should have returned an error")
+	}
+
+	if !s.Visit(cid2) {
+		t.Error("should have visited a new Cid")
+	}
+
+	if s.Len() != 2 {
+		t.Error("len should be 2 now")
+	}
+
+	s.Remove(cid2)
+
+	if s.Len() != 1 {
+		t.Error("len should be 1 now")
+	}
+}

varint.go

@@ -0,0 +1,34 @@
+package cid
+
+// Version of varint function that work with a string rather than
+// []byte to avoid unnecessary allocation
+
+// Copyright 2011 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license as given at https://golang.org/LICENSE
+
+// uvarint decodes a uint64 from buf and returns that value and the
+// number of characters read (> 0). If an error occurred, the value is 0
+// and the number of bytes n is <= 0 meaning:
+//
+// 	n == 0: buf too small
+// 	n  < 0: value larger than 64 bits (overflow)
+// 	        and -n is the number of bytes read
+//
+func uvarint(buf string) (uint64, int) {
+	var x uint64
+	var s uint
+	// we have a binary string so we can't use a range loope
+	for i := 0; i < len(buf); i++ {
+		b := buf[i]
+		if b < 0x80 {
+			if i > 9 || i == 9 && b > 1 {
+				return 0, -(i + 1) // overflow
+			}
+			return x | uint64(b)<<s, i + 1
+		}
+		x |= uint64(b&0x7f) << s
+		s += 7
+	}
+	return 0, 0
+}

varint_test.go

@@ -0,0 +1,22 @@
+package cid
+
+import (
+	"encoding/binary"
+	"testing"
+)
+
+func TestUvarintRoundTrip(t *testing.T) {
+	testCases := []uint64{0, 1, 2, 127, 128, 129, 255, 256, 257, 1<<63 - 1}
+	for _, tc := range testCases {
+		buf := make([]byte, 16)
+		binary.PutUvarint(buf, tc)
+		v, l1 := uvarint(string(buf))
+		_, l2 := binary.Uvarint(buf)
+		if tc != v {
+			t.Errorf("roundtrip failed expected %d but got %d", tc, v)
+		}
+		if l1 != l2 {
+			t.Errorf("length incorrect expected %d but got %d", l2, l1)
+		}
+	}
+}