Buffered vs non buffered file IO using go

In Go, you can use the bufio package to perform buffered I/O on files. Buffered I/O means that data is read from or written to a buffer in memory, rather than reading from or writing to the file directly. This can be more efficient because it reduces the number of system calls and can improve the performance of the program.

Let’s compare I/O operation between buffered IO and without buffer.

Write without buffer

Create file /tmp/test.txt then fill it directly using WriteString function with value some text!\n

package main

import (
	"bufio"
	"io"
	"os"
	"testing"
)

func BenchmarkWriteFile(b *testing.B) {
	for n := 0; n < b.N; n++ {
		f, err := os.Create("/tmp/test.txt")
		if err != nil {
			panic(err)
		}

		for i := 0; i < 100000; i++ {
			f.WriteString("some text!\n")
		}

		f.Close()
	}
}

func BenchmarkWriteFileBuffered(b *testing.B) {
	for n := 0; n < b.N; n++ {
		f, err := os.Create("/tmp/test.txt")
		if err != nil {
			panic(err)
		}

		w := bufio.NewWriter(f)

		for i := 0; i < 100000; i++ {
			w.WriteString("some text!\n")
		}

		w.Flush()
		f.Close()
	}
}

func BenchmarkReadFile(b *testing.B) {
	for n := 0; n < b.N; n++ {
		f, err := os.Open("/tmp/test.txt")
		if err != nil {
			panic(err)
		}

		b := make([]byte, 10)

		_, err = f.Read(b)
		for err == nil {
			_, err = f.Read(b)
		}
		if err != io.EOF {
			panic(err)
		}

		f.Close()
	}
}

func BenchmarkReadFileBuffered(b *testing.B) {
	for n := 0; n < b.N; n++ {
		f, err := os.Open("/tmp/test.txt")
		if err != nil {
			panic(err)
		}

		r := bufio.NewReader(f)

		_, err = r.ReadString('\n')
		for err == nil {
			_, err = r.ReadString('\n')
		}
		if err != io.EOF {
			panic(err)
		}

		f.Close()
	}
}

func BenchmarkReadFileBufferedScanner(b *testing.B) {
	for n := 0; n < b.N; n++ {
		f, err := os.Open("/tmp/test.txt")
		if err != nil {
			panic(err)
		}

		scanner := bufio.NewScanner(f)

		for scanner.Scan() {
			scanner.Text()
		}

		if err := scanner.Err(); err != nil {
			fmt.Fprintln(os.Stderr, "reading standard input:", err)
		}

		f.Close()
	}
}

Write using buffer

Instead of writing directly to file, use bufio.NewWriter(f) to buffer data then flush it to file.

Read without buffer

Reading file without buffer is read chunk byte data until got EOF (end of file)

Read using buffered reader

Read using buffer is similar with writing file using buffer, using r := bufio.NewReader(f) to read all string line by line.

Read using buffered scanner

Using scanner to read all files.

The result from benchmark above is here:

❯ go test -bench=File -benchmem
goos: linux
goarch: amd64
pkg: github.com/h4ckm03d/golang-playground/6-practical-benchmark
cpu: AMD Ryzen 7 5800H with Radeon Graphics
BenchmarkWriteFile-4                          14          80855082 ns/op             120 B/op          3 allocs/op
BenchmarkWriteFileBuffered-4                 438          12895724 ns/op            4216 B/op          4 allocs/op
BenchmarkReadFile-4                           44          25724519 ns/op             120 B/op          3 allocs/op
BenchmarkReadFileBuffered-4                  391           3051139 ns/op         1604219 B/op     100004 allocs/op
BenchmarkReadFileBufferedScanner-4           864           1340817 ns/op            4216 B/op          4 allocs/op
PASS
ok      github.com/h4ckm03d/golang-playground/6-practical-benchmark     11.107s

Based on the benchmark results, it looks like using buffered I/O is generally more efficient than using non-buffered I/O.

In particular, the WriteFileBuffered and ReadFileBufferedScanner benchmarks show significantly better performance than the corresponding non-buffered benchmarks. The ReadFileBuffered benchmark also shows improved performance, but not as significantly as the other two.

It’s worth noting that the ReadFileBuffered benchmark allocates significantly more memory than the other benchmarks, which could be a concern if you are working with large files and need to minimize memory usage. In this case, you might want to consider using the ReadFileBufferedScanner benchmark, which has lower memory usage but still shows improved performance over the non-buffered version.