📖 Advanced File Handle Management — Internals & Optimization

Understand file handles, buffering strategies, and optimize file I/O performance.

🎯 File Handle Internals

Each open file consumes OS resources. Understanding file descriptors is crucial.

import os

# File descriptor is an integer that represents an open file
file1 = open("data.txt", "r")
print(file1.fileno())  # 3 (file descriptor number)

file2 = open("output.txt", "w")
print(file2.fileno())  # 4

# OS limits number of open files
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f"File descriptor limit: {soft}/{hard}")  # Often 256/unlimited on macOS

file1.close()
file2.close()

💡 Buffering Strategies

Buffering affects performance and data safety. Three buffering modes exist:

# Mode 0: Unbuffered (binary mode only)
# Every write goes to disk immediately
with open("data.bin", "wb", buffering=0) as file:
    file.write(b"test")  # Immediate disk write

# Mode 1: Line buffered (text mode default)
# Flushes on newline or when buffer full
with open("data.txt", "w", buffering=1) as file:
    file.write("Line 1\n")  # Flushes immediately
    file.write("No newline")  # Still in buffer

# Mode > 1: Fully buffered (block size)
# Writes in chunks (usually 4KB or 8KB)
with open("data.txt", "w", buffering=8192) as file:
    file.write("x" * 10000)  # Buffered until 8KB

# Benchmark: buffering impact
import timeit

# Unbuffered (slow)
setup_unbuffered = """
with open('test.txt', 'w', buffering=0) as f:
    for i in range(1000):
        f.write(f'Line {i}\\n')
"""

# Fully buffered (fast)
setup_buffered = """
with open('test.txt', 'w', buffering=8192) as f:
    for i in range(1000):
        f.write(f'Line {i}\\n')
"""

🎨 Advanced Handle Management

# Flush manually to ensure data written
with open("data.txt", "w") as file:
    file.write("Important data")
    file.flush()  # Ensure written to disk
    # Process continues even if program crashes

# Isatty: check if connected to terminal
with open("data.txt", "r") as file:
    if file.isatty():
        print("Opened from terminal")
    else:
        print("Opened from file/pipe")

# Seeking in files
with open("data.bin", "rb") as file:
    file.seek(0, 2)  # Go to end
    size = file.tell()  # Get position
    print(f"File size: {size} bytes")

    file.seek(0)  # Go back to start
    data = file.read()  # Read all

📊 Performance Characteristics

🔑 Key Takeaways

• ✅ File descriptors are limited OS resources

• ✅ Buffering dramatically affects performance

• ✅ Use `flush()` for critical data

• ✅ Block buffering fastest for batch operations

• ✅ Monitor file descriptor usage in long-running apps

Ready to practice? Challenges | Quiz