Specify Buffer Size When Opening a File in Python

When we open a file using the Python's built-in function open(), then the data temporarily stored in memory can be managed by setting the buffering parameter in the function. Buffering helps to improve the performance of opening a file by reducing the number of interactions with the disk during file input/output operations.

Understanding the Buffering Parameter

The buffering parameter in Python's open() function allows us to define how much data is stored in memory before being written to or read from the file. This buffering parameter is used to handle file operations with large data or frequent writes.

Syntax

Here is the syntax of the open() function, which uses the Buffering parameter -

open(file, mode='r', buffering=-1, encoding=None, ...)

Buffering Modes

The buffering parameter can take several values, each defining a specific buffering strategy. They are as follows -

• 0: No buffering i.e., only allowed in binary mode. Every read/write operation goes directly to disk.
• 1: Line buffering i.e., only in text mode. The buffer is flushed automatically when a newline character is encountered.
• >1: A buffer of approximately that many bytes is used and helps in reading/writing large files efficiently.
• -1: Use the system's default buffer size, which is the default behavior.

Reading a File with Custom Buffer Size

To read a file with a custom buffer size, we have to pass the desired size in bytes to the buffering argument. In the below example, we are using a 4096-byte buffer, which is more efficient for large files than using the system default -

with open(r"D:\Tutorialspoint\Articles\file1.txt", "r", buffering=4096) as file:
    content = file.read()
    print(content)

Here is the output of the above program -

Hello, welcome to Tutorialspoint.
Have a happy learning.

Binary File with No Buffering

Disabling buffering is useful for binary streams or when working with hardware interfaces that require immediate read/write access. It is only allowed in binary mode. Below is the example in which no buffering for the binary file -

with open(r"D:\Tutorialspoint\Articles\file1.txt", "rb", buffering=0) as file:
    byte = file.read(1)
    print(byte)

Following is the output of the above example -

b'H'

Line Buffering in Text Mode

Line buffering is suitable for log files or output that must be written immediately after each line. In this mode, the data is flushed to disk as soon as a newline character is written by ensuring that log entries are saved in real time. Below is an example which shows Line Buffering in Text Mode -

with open(r"D:\Tutorialspoint\Articles\file1.txt", "w", buffering=1) as file:
    file.write("Log entry: Application started\n")

Default Buffer Size

If we do not specify a buffer size or use buffering=-1, then Python will automatically choose a buffer size based on the platform and file type ?

with open("default.txt", "r", buffering=-1) as file:
    data = file.read()

When to Customize the Buffer Size?

Even Python handles buffering by default, there are certain situations where specifying a custom buffer size is beneficial, as follows -

• Large file operations: Increase buffer size to reduce the number of I/O calls and improve performance.
• Real-time output: Use line buffering to ensure immediate write to disk e.g., in logging.
• Precision control: Disable buffering for hardware interaction or byte-by-byte communication.