A Primer on Go's bufio Package

If you have been using Go even for just a few months, you probably have come across the bufio package. The bufio package implements buffered I/O.

What does that mean?

Understanding buffered I/O

A buffer refers to a temporary storage area used for efficient I/O operations. It is essentially a space in memory that holds data before it is read from or written to an I/O device.

An I/O device may be a file, network connection, or standard input/output.

Buffers allow us to reduce the number of direct I/O operations by collecting or distributing data in larger chunks rather than individually processing each byte or character.

A system call is required whenever you want to write to a file. So if you directly write to a file 100 times, 100 system calls will be made. This is overly simplified, of course.

With buffered I/O, that number can be significantly reduced.

In this case, the contents of the buffer are written to the file only when the buffer is full.

The Reader and Writer interface

The io package provides the Reader and Writer interfaces, which are abstractions for generically handling input and output operations irrespective of the data source or destination.

The io.Reader interface represents an object that can be read from. It has a single method.

func (T) Read(b []byte) (n int, err error)

This method reads data into the provided byte slice b and returns the number of bytes read - n and an error, err.

This makes it possible to read from various sources like files, network connections, strings, and more by implementing the Read method.

The io.Writer interface represents an object that can be written to. It also consists of a single method.

func (T) Write(b []byte) (n int, err error)

The Write method takes a byte slice b, and writes its content to the underlying data sink, which could be a file, network connection, or others, returning the number of bytes(n) written and an error(err)

Various types in Go implement this interface including os.File, bytes.Buffer and bufio.Writer

Reading with bufio

Example: Reading n amount of bytes from a file

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    b, err := readNAmountOfBytes("test.txt", 10)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(b))
}

func readNAmountOfBytes(filepath string, n int) ([]byte, error) {
    f, err := os.Open(filepath)
    if err != nil {
        return nil, err
    }
    defer f.Close()
    reader := bufio.NewReaderSize(f, n)
    var b []byte = make([]byte, n)
    reader.Read(b)
    return b, nil
}

We've introduced the NewReaderSize function.

It creates a buffered reader that reads from an io.Reader (such as a file or network connection) with a specified buffer size.

This is especially useful for scenarios where you want to control the buffer size based on the expected data size or to optimize I/O operations by reading data in specified chunk sizes.

Example: Reading an entire file's content

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    b, err := readEntireContent("test.txt")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(b))
}

func readEntireContent(filepath string) (b []byte, err error) {
    f, err := os.Open(filepath)
    if err != nil {
        return nil, err
    }
    defer f.Close()
    reader := bufio.NewScanner(f)
    for reader.Scan() {
        b = append(b, reader.Bytes()...)
        b = append(b, []byte("\n")...) // need to preserve new lines in text
    }
    return
}

This example introduces another function in the bufio package - bufio.NewScanner

This function returns a Scanner type that efficiently tokenizes input, allowing iteration through the input data in convenient chunks or lines, depending on the defined split function.

By default, the scanner splits the input into lines('\n') and provides methods like Scan() to advance through the input and retrieve tokens as byte slices.

Example: Reading from standard input

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    input := readFromStdIn("Enter your age: ")
    fmt.Println(input)
}

func readFromStdIn(prompt string) (text string) {
    scanner := bufio.NewScanner(os.Stdin)
    fmt.Print(prompt)
    s := scanner.Scan()
    if s {
        text = scanner.Text()
    }
    return
}

readFromStdin does the following

  • Accepts a prompt string parameter that represents the message to display.

  • Creates a Scanner using bufio.NewScanner(os.Stdin) to read from the standard input (os.Stdin).

  • Displays the prompt using fmt.Print(prompt) to ask the user for input.

  • Invokes scanner.Scan() to read a line from the standard input. It returns a boolean indicating whether scanning succeeded or not.

  • If scanning succeeded (scanner.Scan() returned true), it retrieves the text entered by the user using scanner.Text() and assigns it to the text variable.

Example: Creating your own Reader

As mentioned earlier, a Reader implements the io.Reader interface which contains a single method.

func (T) Read(b []byte) (n int, err error)
package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

// MyReader is a custom reader type with data and a position tracker.
type MyReader struct {
    data string
    pos  int
}

// Read implements the Reader interface for MyReader.
// It reads data from MyReader into the provided byte slice.
func (r *MyReader) Read(p []byte) (n int, err error) {
    // If the position is at or beyond the length of data, return EOF.
    if r.pos >= len(r.data) {
        return 0, io.EOF
    }
    // Copy the data from r.data starting from r.pos into p.
    n = copy(p, r.data[r.pos:])
    // Update the position after the copy operation.
    r.pos += n
    return
}

func main() {
    // Create an instance of MyReader with the initial data "Hello, world!" and position 0.
    r := MyReader{"Hello, world!", 0}
    n := 10 // Define the number of bytes to read
    data := make([]byte, n) // Create a byte slice of length n to store read data
    readFromMyReader(data, n, &r) // Read data from MyReader into the data byte slice
    fmt.Println(string(data)) // Print the data read from MyReader
}

// readFromMyReader reads data from MyReader into the provided byte slice.
func readFromMyReader(dst []byte, n int, r *MyReader) (err error) {
    // Create a buffered reader with the size n and the provided MyReader instance.
    reader := bufio.NewReaderSize(r, n)
    // Read data from the buffered reader into the dst byte slice.
    _, err = reader.Read(dst)
    return
}

Writing with bufio

In the next sections, we will look at different ways to read using the bufio package.

Example: Writing to a file

package main

import (
    "bufio"
    "log"
    "os"
)

func main() {
    // Calls the writeToFile function to write "hello world" to a file named "output.txt"
    err := writeToFile("output.txt", "hello world")
    if err != nil {
        // If an error occurred during file write, log the error and exit the program
        log.Fatal(err)
    }
}

// writeToFile function takes a filename and data as input and writes the data to the specified file.
func writeToFile(filename string, data string) error {
    // Open or create the file with write-only permissions and a file mode of 0644
    f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        // If an error occurs while opening or creating the file, return the error
        return err
    }
    defer f.Close() // Close the file when the function returns

    // Create a buffered writer that writes to the opened file
    writer := bufio.NewWriter(f)

    // Write the provided data to the buffered writer
    _, err = writer.WriteString(data)
    if err != nil {
        // If an error occurs while writing to the file, return the error
        return err
    }

    // Flush any buffered data to ensure it's written to the file
    return writer.Flush()
}

Example: Writing to standard output

package main

import (
    "bufio"
    "log"
    "os"
)

func main() {
    // Calls the writeToStdOut function to write "writing to standard output" to the console
    err := writeToStdOut("writing to standard output")
    if err != nil {
        // If an error occurred during the write operation, log the error and exit the program
        log.Fatal(err)
    }
}

// writeToStdOut function writes the provided data to the standard output (console)
func writeToStdOut(data string) error {
    // Create a buffered writer that writes to the standard output
    writer := bufio.NewWriter(os.Stdout)

    // Write the provided data to the buffered writer (which is the standard output)
    _, err := writer.WriteString(data)
    if err != nil {
        // If an error occurs while writing to the standard output, return the error
        return err
    }

    // Flush any buffered data to ensure it's written to the standard output
    return writer.Flush()
}

The default buffer size is 4096 bytes.

If the data we are writing is less than that size, it will remain in the buffer and not drained into the data sink(file).

That is why a call to writer.Flush() is necessary.

We can use the bufio.NewWriterSize(w io.Writer, size int) if we want to change the default.

Example: Creating your own Writer

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
    "strings"
)

// CustomWriter is a struct that will implement the io.Writer interface.
type CustomWriter struct {
    // You can include any additional fields needed for your custom writer
    // (if necessary)
}

// Write implements the io.Writer interface for CustomWriter.
// This method takes incoming bytes and capitalizes them before writing.
func (cw *CustomWriter) Write(p []byte) (n int, err error) {
    // Capitalize incoming bytes and write them to standard output
    capitalized := strings.ToUpper(string(p))
    fmt.Print(capitalized)
    return len(p), nil
}

func main() {
    // Create an instance of CustomWriter
    cw := &CustomWriter{}

    // Create a buffered writer that uses the CustomWriter
    writer := bufio.NewWriter(cw)

    // Use the buffered writer to write some data
    _, _ = writer.WriteString("writing with a custom writer\n")
    _, _ = writer.WriteString("this is so cool!\n")

    // Flush any remaining data in the buffer to the CustomWriter
    _ = writer.Flush()
}

That's it for now. I hope you found this article helpful :)