The devil is in the detail!

I was writing a Go program this morning, and I found myself in need to move all elements from a map to a slice. Easy task, right? Kind of.

The solution is easy, but it's even easier to make a subtle mistake in it, so I decided to write a blog post about the possible mistakes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
type Tx struct {
	from  string
	to    string
	value uint64
}

txMap := make(map[string]Tx)
txMap["tx1"] = Tx{"andrej", "babayaga", 10}
txMap["tx2"] = Tx{"andrej", "babayaga", 20}
txMap["tx3"] = Tx{"andrej", "babayaga", 30}

// CONVERT ME TO A SLICE

Gotcha 1 - Array != Slice

An experienced gopher knows this by the hearth, but if you are a developer who jumps between languages depending on the project, it may get confusing.

You need to understand the difference between an array and a slice to avoid Gotcha 2 and 3. I recommend the official Slices introduction from the Go team.

Long story short:

  • array has a fixed length: var txArray [3]Tx
  • slice is a resizable abstraction with extra features and a pointer to a fixed-length array in the background

Gotcha 2 - Slice with a 0 default length

The most common way to achieve this is by creating a new slice with 0 length txs := make([]Tx, 0) and populate the array behind the scenes using the slice's append function txs = append(txs, tx).

Full example:

package main

import (
	"github.com/davecgh/go-spew/spew"
)

type Tx struct {
	from  string
	to    string
	value uint64
}

func main() {
	txMap := make(map[string]Tx)
	txMap["tx1"] = Tx{"andrej", "babayaga", 10}
	txMap["tx2"] = Tx{"andrej", "babayaga", 20}
	txMap["tx3"] = Tx{"andrej", "babayaga", 30}

	txSlice := getTXsAsSliceByReallocatingMemory(txMap)
	spew.Dump(txSlice)
}

func getTXsAsSliceByReallocatingMemory(txMap map[string]Tx) []Tx {
	// 0 default length is not performance wise
	txs := make([]Tx, 0)

	for _, tx := range txMap {
		txs = append(txs, tx)
	}

	return txs
}

While this technically works, it's inefficient because behind the scenes - Go creates an array with a fixed length. When this array gets full of new elements coming in from the append() function, Go creates another array with double the memory copy all the elements into the new array. This repeats until necessary.

Appending 3 elements into an empty slice causes Go to allocate memory for 3 arrays with lengths (1, 2, 4) and re-copy their content whenever the memory doubles.

While this is not a problem with few elements, keep it in mind when working with large datasets.

Inefficient result:

([]main.Tx) (len=3 cap=4) {
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 10
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 20
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 30
 }
}
					

Gotcha 3 - Slice with len(txMap)

No problem! I initialize a slice with the length of the map from where I want to transfer the elements!

txs := make([]Tx, len(txMap))

Wrong!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
package main

import (
	"github.com/davecgh/go-spew/spew"
)

type Tx struct {
	from  string
	to    string
	value uint64
}

func main() {
	txMap := make(map[string]Tx)
	txMap["tx1"] = Tx{"andrej", "babayaga", 10}
	txMap["tx2"] = Tx{"andrej", "babayaga", 20}
	txMap["tx3"] = Tx{"andrej", "babayaga", 30}

	txSlice := getTXsAsSliceByReallocatingMemory(txMap)
	spew.Dump(txSlice)
}

func getTXsAsSliceByReallocatingMemory(txMap map[string]Tx) []Tx {
	// Oooops, this didn't work out as planned
	txs := make([]Tx, len(txMap))

	for _, tx := range txMap {
		txs = append(txs, tx)
	}

	return txs
}

Results in 3 extra empty elements:

([]main.Tx) (len=6 cap=6) {
 (main.Tx) {
  from: (string) "", to: (string) "", value: (uint64) 0
 },
 (main.Tx) {
  from: (string) "", to: (string) "", value: (uint64) 0
 },
 (main.Tx) {
  from: (string) "", to: (string) "", value: (uint64) 0
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 10
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 20
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 30
 }
}
					

Efficient solutions

1) Define Slice capacity

Using the third argument of the make function.

txs := make([]Tx, 0, len(txMap))

Thanks for the suggestion Jakub Gawlas.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
package main

import (
	"github.com/davecgh/go-spew/spew"
)

type Tx struct {
	from  string
	to    string
	value uint64
}

func main() {
	// Extra touch pre-defining the Map length to avoid reallocation
	txMap := make(map[string]Tx, 3)
	txMap["tx1"] = Tx{"andrej", "babayaga", 10}
	txMap["tx2"] = Tx{"andrej", "babayaga", 20}
	txMap["tx3"] = Tx{"andrej", "babayaga", 30}

	txSlice := getTXsAsSlice(txMap)
	spew.Dump(txSlice)
}

func getTXsAsSlice(txMap map[string]Tx) []Tx {
	// Defines the Slice capacity to match the Map elements count
	txs := make([]Tx, 0, len(txMap))
	for _, tx := range txMap {
		txs = append(txs, tx)
	}

	return txs
}

2) Or use an old-fashion loop counter

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package main

import (
	"github.com/davecgh/go-spew/spew"
)

type Tx struct {
	from  string
	to    string
	value uint64
}

func main() {
	// Extra touch pre-defining the Map length to avoid reallocation
	txMap := make(map[string]Tx, 3)
	txMap["tx1"] = Tx{"andrej", "babayaga", 10}
	txMap["tx2"] = Tx{"andrej", "babayaga", 20}
	txMap["tx3"] = Tx{"andrej", "babayaga", 30}

	txSlice := getTXsAsSlice(txMap)
	spew.Dump(txSlice)
}

func getTXsAsSlice(txMap map[string]Tx) []Tx {
	// Defines the Slice length to match the Map elements count
	txs := make([]Tx, len(txMap))

	i := 0
	for _, tx := range txMap {
		txs[i] = tx
		i++
	}

	return txs
}

Both create a 3 elements long slice right away without any overhead.

Result:

([]main.Tx) (len=3 cap=3) {
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 10
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 20
 },
 (main.Tx) {
  from: (string) (len=6) "andrej", to: (string) (len=8) "babayaga", value: (uint64) 30
 }
}
					

Thanks for reading!

Get Started with Blockchain Development

Hi, I am writing an eBook on how to build a peer-to-peer system in Go from scratch!

Follow how the main protagonist Andrej, a web developer by day and a bartender by night, learns real-world use of blockchain through the growth of his new business venture: "The Blockchain Bar".

Free chapter