📖 Advanced Reading Strategies — Streaming & Memory Efficiency

Master efficient reading patterns for files of any size, from MB to GB+.

🎯 Generator-Based Reading for Memory Efficiency

Generators allow processing files without loading entire contents into memory.

def read_large_file(file_path, chunk_size=8192):
    """Read file in chunks without loading all at once"""
    with open(file_path, "rb") as file:
        while True:
            chunk = file.read(chunk_size)
            if not chunk:
                break
            yield chunk

# Usage: Process gigabyte files without memory spike
for chunk in read_large_file("large_file.bin"):
    process(chunk)  # Handle 8KB at a time

💡 Advanced Line Reading Patterns

def read_lines_optimized(file_path, buffer_size=65536):
    """Read lines efficiently with custom buffering"""
    with open(file_path, "rb") as file:
        buffer = b""
        while True:
            data = file.read(buffer_size)
            if not data:
                if buffer:
                    yield buffer.decode("utf-8")
                break

            buffer += data
            lines = buffer.split(b"\n")
            buffer = lines[-1]  # Keep incomplete line

            for line in lines[:-1]:
                yield line.decode("utf-8")

# Process multi-gigabyte logs line by line
for line_num, line in enumerate(read_lines_optimized("huge.log")):
    if "ERROR" in line:
        print(f"Error at line {line_num}: {line}")

🎨 Memory-Mapped Files for Random Access

import mmap

# Memory-map binary file for efficient random access
with open("data.bin", "rb") as file:
    with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mmapped:
        # Access file as if it were in memory
        chunk = mmapped[100:200]  # Fast random access
        index = mmapped.find(b"pattern")  # Search efficiently

# Performance: 1000x faster than repeated seek/read for random access
import timeit

# Traditional approach
setup_traditional = """
with open('large.bin', 'rb') as f:
    for i in range(0, 1000000, 10000):
        f.seek(i)
        data = f.read(100)
"""

# Memory-mapped approach
setup_mmap = """
import mmap
with open('large.bin', 'rb') as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m:
        for i in range(0, 1000000, 10000):
            data = m[i:i+100]
"""

📊 Reading Performance Comparison

🔑 Key Takeaways

• ✅ Generators for streaming large files

• ✅ Memory-mapped files for random access

• ✅ Chunked reading prevents memory spikes

• ✅ Custom buffering for optimization

• ✅ Choose strategy based on access pattern

Ready to practice? Challenges | Quiz