If you have been using Go even for just a few months, you probably have come across the bufio
package. The bufio
package implements buffered I/O.
What does that mean?
Understanding buffered I/O
A buffer refers to a temporary storage area used for efficient I/O operations. It is essentially a space in memory that holds data before it is read from or written to an I/O device.
An I/O device may be a file, network connection, or standard input/output.
Buffers allow us to reduce the number of direct I/O operations by collecting or distributing data in larger chunks rather than individually processing each byte or character.
A system call is required whenever you want to write to a file. So if you directly write to a file 100 times, 100 system calls will be made. This is overly simplified, of course.
With buffered I/O, that number can be significantly reduced.
In this case, the contents of the buffer are written to the file only when the buffer is full.
The Reader
and Writer
interface
The io
package provides the Reader
and Writer
interfaces, which are abstractions for generically handling input and output operations irrespective of the data source or destination.
The io.Reader
interface represents an object that can be read from. It has a single method.
func (T) Read(b []byte) (n int, err error)
This method reads data into the provided byte slice b and returns the number of bytes read - n
and an error, err
.
This makes it possible to read from various sources like files, network connections, strings, and more by implementing the Read
method.
The io.Writer
interface represents an object that can be written to. It also consists of a single method.
func (T) Write(b []byte) (n int, err error)
The Write
method takes a byte slice b
, and writes its content to the underlying data sink, which could be a file, network connection, or others, returning the number of bytes(n
) written and an error(err
)
Various types in Go implement this interface including os.File
, bytes.Buffer
and bufio.Writer
Reading with bufio
Example: Reading n
amount of bytes from a file
package main
import (
"bufio"
"fmt"
"log"
"os"
)
func main() {
b, err := readNAmountOfBytes("test.txt", 10)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
func readNAmountOfBytes(filepath string, n int) ([]byte, error) {
f, err := os.Open(filepath)
if err != nil {
return nil, err
}
defer f.Close()
reader := bufio.NewReaderSize(f, n)
var b []byte = make([]byte, n)
reader.Read(b)
return b, nil
}
We've introduced the NewReaderSize
function.
It creates a buffered reader that reads from an io.Reader
(such as a file or network connection) with a specified buffer size.
This is especially useful for scenarios where you want to control the buffer size based on the expected data size or to optimize I/O operations by reading data in specified chunk sizes.
Example: Reading an entire file's content
package main
import (
"bufio"
"fmt"
"log"
"os"
)
func main() {
b, err := readEntireContent("test.txt")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
func readEntireContent(filepath string) (b []byte, err error) {
f, err := os.Open(filepath)
if err != nil {
return nil, err
}
defer f.Close()
reader := bufio.NewScanner(f)
for reader.Scan() {
b = append(b, reader.Bytes()...)
b = append(b, []byte("\n")...) // need to preserve new lines in text
}
return
}
This example introduces another function in the bufio
package - bufio.NewScanner
This function returns a Scanner
type that efficiently tokenizes input, allowing iteration through the input data in convenient chunks or lines, depending on the defined split function.
By default, the scanner splits the input into lines('\n')
and provides methods like Scan()
to advance through the input and retrieve tokens as byte slices.
Example: Reading from standard input
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
input := readFromStdIn("Enter your age: ")
fmt.Println(input)
}
func readFromStdIn(prompt string) (text string) {
scanner := bufio.NewScanner(os.Stdin)
fmt.Print(prompt)
s := scanner.Scan()
if s {
text = scanner.Text()
}
return
}
readFromStdin
does the following
Accepts a
prompt
string parameter that represents the message to display.Creates a
Scanner
usingbufio.NewScanner(os.Stdin)
to read from the standard input (os.Stdin
).Displays the prompt using
fmt.Print(prompt)
to ask the user for input.Invokes
scanner.Scan()
to read a line from the standard input. It returns a boolean indicating whether scanning succeeded or not.If scanning succeeded (
scanner.Scan()
returnedtrue
), it retrieves the text entered by the user usingscanner.Text()
and assigns it to thetext
variable.
Example: Creating your own Reader
As mentioned earlier, a Reader
implements the io.Reader
interface which contains a single method.
func (T) Read(b []byte) (n int, err error)
package main
import (
"bufio"
"fmt"
"io"
"os"
)
// MyReader is a custom reader type with data and a position tracker.
type MyReader struct {
data string
pos int
}
// Read implements the Reader interface for MyReader.
// It reads data from MyReader into the provided byte slice.
func (r *MyReader) Read(p []byte) (n int, err error) {
// If the position is at or beyond the length of data, return EOF.
if r.pos >= len(r.data) {
return 0, io.EOF
}
// Copy the data from r.data starting from r.pos into p.
n = copy(p, r.data[r.pos:])
// Update the position after the copy operation.
r.pos += n
return
}
func main() {
// Create an instance of MyReader with the initial data "Hello, world!" and position 0.
r := MyReader{"Hello, world!", 0}
n := 10 // Define the number of bytes to read
data := make([]byte, n) // Create a byte slice of length n to store read data
readFromMyReader(data, n, &r) // Read data from MyReader into the data byte slice
fmt.Println(string(data)) // Print the data read from MyReader
}
// readFromMyReader reads data from MyReader into the provided byte slice.
func readFromMyReader(dst []byte, n int, r *MyReader) (err error) {
// Create a buffered reader with the size n and the provided MyReader instance.
reader := bufio.NewReaderSize(r, n)
// Read data from the buffered reader into the dst byte slice.
_, err = reader.Read(dst)
return
}
Writing with bufio
In the next sections, we will look at different ways to read using the bufio
package.
Example: Writing to a file
package main
import (
"bufio"
"log"
"os"
)
func main() {
// Calls the writeToFile function to write "hello world" to a file named "output.txt"
err := writeToFile("output.txt", "hello world")
if err != nil {
// If an error occurred during file write, log the error and exit the program
log.Fatal(err)
}
}
// writeToFile function takes a filename and data as input and writes the data to the specified file.
func writeToFile(filename string, data string) error {
// Open or create the file with write-only permissions and a file mode of 0644
f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
// If an error occurs while opening or creating the file, return the error
return err
}
defer f.Close() // Close the file when the function returns
// Create a buffered writer that writes to the opened file
writer := bufio.NewWriter(f)
// Write the provided data to the buffered writer
_, err = writer.WriteString(data)
if err != nil {
// If an error occurs while writing to the file, return the error
return err
}
// Flush any buffered data to ensure it's written to the file
return writer.Flush()
}
Example: Writing to standard output
package main
import (
"bufio"
"log"
"os"
)
func main() {
// Calls the writeToStdOut function to write "writing to standard output" to the console
err := writeToStdOut("writing to standard output")
if err != nil {
// If an error occurred during the write operation, log the error and exit the program
log.Fatal(err)
}
}
// writeToStdOut function writes the provided data to the standard output (console)
func writeToStdOut(data string) error {
// Create a buffered writer that writes to the standard output
writer := bufio.NewWriter(os.Stdout)
// Write the provided data to the buffered writer (which is the standard output)
_, err := writer.WriteString(data)
if err != nil {
// If an error occurs while writing to the standard output, return the error
return err
}
// Flush any buffered data to ensure it's written to the standard output
return writer.Flush()
}
The default buffer size is 4096 bytes.
If the data we are writing is less than that size, it will remain in the buffer and not drained into the data sink(file).
That is why a call to
writer.Flush()
is necessary.
We can use the bufio.NewWriterSize(w io.Writer, size int)
if we want to change the default.
Example: Creating your own Writer
package main
import (
"bufio"
"fmt"
"io"
"os"
"strings"
)
// CustomWriter is a struct that will implement the io.Writer interface.
type CustomWriter struct {
// You can include any additional fields needed for your custom writer
// (if necessary)
}
// Write implements the io.Writer interface for CustomWriter.
// This method takes incoming bytes and capitalizes them before writing.
func (cw *CustomWriter) Write(p []byte) (n int, err error) {
// Capitalize incoming bytes and write them to standard output
capitalized := strings.ToUpper(string(p))
fmt.Print(capitalized)
return len(p), nil
}
func main() {
// Create an instance of CustomWriter
cw := &CustomWriter{}
// Create a buffered writer that uses the CustomWriter
writer := bufio.NewWriter(cw)
// Use the buffered writer to write some data
_, _ = writer.WriteString("writing with a custom writer\n")
_, _ = writer.WriteString("this is so cool!\n")
// Flush any remaining data in the buffer to the CustomWriter
_ = writer.Flush()
}
That's it for now. I hope you found this article helpful :)