Poor performance on logging.RotatingFileHandler due to fix to #89564

[zhatt]

The fix to #89564 produced a significant performance degradation when logs are on an NFS filesystem. The fix for #89564 was for a bug found with the TimedRotatingFileHandler but an fix was added to the sized base file rotator too.

Here is the RotatingFileHandler.shouldRollover() function. The os.path.exists() and os.path.isfile() are very slow on NFS filesystems. On Linux systems, if os.path.isfile() returns False, the self.stream.seek(0,2) will always return 0 because the file is not seekable so non-isfile() files will never be rotated unless the rotation size is smaller than the message size.

Since, the rollover could be triggered with a very large msg and small maxBytes. Moving the exists() and isfile() check to inside the if self.stream.tell()... would cover that case and not run the expensive status operations with a correctly configured logger except when the rotation is needed. That is similar to the fix made to the TimedRotatingFileHandler in #96159.

    def shouldRollover(self, record):
        """
        Determine if rollover should occur.

        Basically, see if the supplied record would cause the file to exceed
        the size limit we have.
        """
        # See bpo-45401: Never rollover anything other than regular files
        if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename):
            return False
        if self.stream is None:                 # delay was set...
            self.stream = self._open()
        if self.maxBytes > 0:                   # are we rolling over?
            msg = "%s\n" % self.format(record)
            self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
            if self.stream.tell() + len(msg) >= self.maxBytes:
                return True
        return False

Linked PRs

• gh-105623 Fix performance degradation in logging RotatingFileHandler #105887
• [3.12] gh-105623 Fix performance degradation in logging RotatingFileHandler (GH-105887) #121116
• [3.13] gh-105623 Fix performance degradation in logging RotatingFileHandler (GH-105887) #121117

[arhadthedev]

cc @vstinner as an author of the linked issue gh-89564 and @vsajip as an author of its associated PR.

[vstinner]

Do you want to propose a fix? (Pull request)

[zhatt]

@vstinner I will create a PR.