Comprehensive Guide to File Reading and Writing in Go: From Basics to Advanced Practices

Introduction

File operations are fundamental in any programming language, and Go provides multiple file reading and writing approaches through its concise yet powerful standard library. Based on Go 1.x versions, this article systematically introduces core methods for file operations, helping developers understand best practices for different scenarios.

Basic File Operations

Go uses the os package for basic file operations. os.Open is used to open existing files, while os.Create creates new files. Both functions are convenient wrappers around os.OpenFile, and in most cases, there's no need to call the underlying function directly.

The following example demonstrates file copying using basic methods:

package main

import (
    "io"
    "os"
)

func main() {
    // Open input file
    fi, err := os.Open("input.txt")
    if err != nil {
        panic(err)
    }
    defer func() {
        if err := fi.Close(); err != nil {
            panic(err)
        }
    }()

    // Create output file
    fo, err := os.Create("output.txt")
    if err != nil {
        panic(err)
    }
    defer func() {
        if err := fo.Close(); err != nil {
            panic(err)
        }
    }()

    // Create buffer for reading data chunks
    buf := make([]byte, 1024)
    for {
        n, err := fi.Read(buf)
        if err != nil && err != io.EOF {
            panic(err)
        }
        if n == 0 {
            break
        }

        if _, err := fo.Write(buf[:n]); err != nil {
            panic(err)
        }
    }
}

Key points analysis:

• defer statements ensure files are properly closed when function exits
• Buffer size affects read/write performance, 1024 bytes is a common choice
• io.EOF error indicates end of file and requires special handling
• The returned byte count n from reading must be used for writing to avoid writing excess data

Buffered Reading and Writing with bufio

The bufio package provides buffered I/O capabilities that significantly improve performance for frequent small data reads and writes. It wraps basic file operations and offers a more user-friendly API.

package main

import (
    "bufio"
    "io"
    "os"
)

func main() {
    fi, err := os.Open("input.txt")
    if err != nil {
        panic(err)
    }
    defer func() {
        if err := fi.Close(); err != nil {
            panic(err)
        }
    }()

    r := bufio.NewReader(fi)

    fo, err := os.Create("output.txt")
    if err != nil {
        panic(err)
    }
    defer func() {
        if err := fo.Close(); err != nil {
            panic(err)
        }
    }()

    w := bufio.NewWriter(fo)

    buf := make([]byte, 1024)
    for {
        n, err := r.Read(buf)
        if err != nil && err != io.EOF {
            panic(err)
        }
        if n == 0 {
            break
        }

        if _, err := w.Write(buf[:n]); err != nil {
            panic(err)
        }
    }

    if err = w.Flush(); err != nil {
        panic(err)
    }
}

Advantages of bufio:

• Automatic buffer management reduces system call frequency
• Provides rich read/write methods like ReadString, ReadBytes, etc.
• Write operations require calling Flush to ensure data is written to disk
• Particularly suitable for line-by-line processing of text files

Convenient File Operations

For small files, convenient one-shot read/write methods can be used. Note that since Go 1.16, the ioutil package has been deprecated, with related functions migrated to os and io packages.

Modern Go versions recommend using:

package main

import "os"

func main() {
    // Read entire file
    data, err := os.ReadFile("input.txt")
    if err != nil {
        panic(err)
    }

    // Write entire file
    err = os.WriteFile("output.txt", data, 0644)
    if err != nil {
        panic(err)
    }
}

File permission explanations:

• 0644 indicates owner can read/write, other users read-only
• 0777 indicates all users have read, write, and execute permissions
• Permission settings should be chosen carefully based on actual security requirements

Best Practices for Error Handling

Go emphasizes explicit error handling. Common errors in file operations include:

• File does not exist
• Insufficient permissions
• Insufficient disk space
• Network file system timeouts

Recommended structured error handling:

func copyFile(src, dst string) error {
    input, err := os.Open(src)
    if err != nil {
        return fmt.Errorf("failed to open source file: %w", err)
    }
    defer input.Close()

    output, err := os.Create(dst)
    if err != nil {
        return fmt.Errorf("failed to create destination file: %w", err)
    }
    defer output.Close()

    _, err = io.Copy(output, input)
    if err != nil {
        return fmt.Errorf("failed to copy file content: %w", err)
    }

    return output.Close()
}

Performance Optimization Recommendations

Choose appropriate read/write strategies based on file size and access patterns:

• Small files: Use os.ReadFile/os.WriteFile for one-shot operations
• Large files: Use buffered reading/writing to avoid memory overflow
• Frequent reads/writes: Use bufio to reduce system calls
• Network files: Consider using larger buffers to reduce network round trips

Conclusion

Go provides multi-level file operation APIs, from basic byte-level operations to advanced buffered reading and writing, capable of meeting requirements across different scenarios. Understanding the applicable scenarios and performance characteristics of these methods helps in writing efficient and reliable file processing code. In practical development, choose the most suitable file operation method based on specific requirements and always follow good error handling practices.