Add `rmtree` & `copy` method to pathlib

[Conchylicultor]

Feature or enhancement

Currently pathlib is missing some shutil features: rmtree or copy

Pitch

pathlib.Path define an Interface / API that many third party modules implement:

• zipfile.Path
• importlib.resource.Transversable
• epath.Path
• universal_pathlib
• gitpathlib.GitPath
• Many others: https://pypi.org/search/?q=pathlib

Having a consistent API is great as I can write code for pathlib, and it is automatically compatible with all pathlib-like backends without any code changes.
(e.g. user can replace pathlib.Path by epath.Path and the function magically work with cloud storage likegs://, s3://,...).

However due to the lack of .rmtree and .copy supports, some operations are impossible. It means each pathlib-like backend who want to support recursive directory deletion or copy has to come up with their own API, leading to inconsistency.

(technically it is possible to implement some rmtree(path.Path()) implemented only using pathlib function, but this would lead in very inefficient performance on remote storage)

Defining them directly in pathlib would make them part of the standard, with well defined semantic. This would allow implementations to rely on them and to switch between pathlib-like objects.

This will be even more relevant when pathlib will provide an explicit interface that users an inherit from: https://discuss.python.org/t/make-pathlib-extensible/3428/7

[zmievsa]

rmtree seems entirely possible, considering we will have walk() soon. I'll have to benchmark to see if we can use my walk() implementation for this (as it can be a bit slow due to the use of stat()).

Considering we have rename(), copy seems to be possible as well.

We could also just wrap shutil's functions in pathlib but then Path._scandir will not be utilized; also shutil is not currently used in pathlib at all -- and we might have a good reason for it.

I believe that @barneygale would also be interested in this conversation.

[zmievsa]

https://discuss.python.org/t/add-pathlib-path-rmtree/15806

[domdfcoding]

I recall a discussion (although despite searching I can't find it) that pathlib shouldn't replicate (or wrap or otherwise provide) this sort of functionality, since the version in shutil is fine. The intention was solely to provide an alternative to os.path et al.

I sort of agree, but I do think there is a compelling case for the two methods you're proposing in the context of Barney's abstract path concept.

[zmievsa]

@domdfcoding I feel like pathlib can and should support most functionality of both os.path and shutil. It feels natural to use rmtree, rmdir, unlink, walk, etc on Path objects.

[brettcannon]

To be clear, pathlib is not expected to fully supplant everything in os or shutil; they exist separately for a reason. It's going to be a per-feature use-case.

[brettcannon]

And to add on to my last comment, each addition needs to somehow improve add to the situation, else it's just wrapping a function call in a method and that isn't helpful enough to need to support that API for decades.

/cc @barneygale

[barneygale]

else it's just wrapping a function call in a method and that isn't helpful enough to need to support that API for decades.

Wrapping a function call in a method is 50% of pathlib, and it made pathlib wildly popular even before it was added to the standard library. The helpfulness comes from the object-oriented API and the DSL-like goodness of constructing paths with /. And I'm hoping to introduce an AbstractPath in future that will allow the likes of cloudpathlib to provide their own implementations, further increasing the utility.

I'd consider the following shutil functions candidates for wrapping in pathlib: move(), copy(), copytree(), rmtree(), chown(). In each case they have similar utility to existing pathlib methods like symlink_to() or rmdir().

@Conchylicultor do you think you could log a separate issue for copy(), and reduce this one to only covering rmtree()? Thanks

[zmievsa]

@brettcannon could you explain what reason there is for pathlib to not support, for example, a majority of functionality of shutil? I've been wondering about that for a while now.

[merwok]

In my recollection, pathlib’s goal is to provide a class abstraction to replace os.path modules, but it is not meant to encompass all operations that work with files in the stdlib. That is why the fspath protocol was added and many modules gained support for strings or Paths. So IMO it’s good and clean if you can call shutil.rmtree(some_path), but there is no strong need to add some_path.rmtree().

[zmievsa]

Good point. I still believe that reading functionality should be a part of pathlib (such as walk) but it makes sense that things like rmtree are not exactly necessary there.

[brettcannon]

@barneygale You're right, making an object-oriented API that cleverly leveraged operator overloading is what has made pathlib successful. But there is a worry of scope creep and trying to keep the API small enough to fit in your head. For instance, .copy() and even more so for copytree() doesn't add anything as you are not manipulating the path object, you're just performing an action on it; the __fspath__ protocol makes this easily compatible with pathlib objects. So the question becomes how does the method add convenience for most users? (I mention your AbstractPath later 😉.)

In all of this, you have to also understand that pathlib quite possibly would not have a lot of what's in there today as it predates __fspath__. For instance, .open() quite possibly wouldn't exist on pathlib.Path since open() itself now natively supports pathlib objects. So things that are not directly related to path manipulation or don't somehow have an added benefit to being on pathlib (e.g. returning a pathlib object itself) very well wouldn't make sense today.

Now I fully admit this is all subjective, but this is what we have to think through with all of this. Remember, any API we add will need to be supported for literally decades and I have to assume I will be the one doing the support (I have personally been on Python's development team for 19 years and I still have code from my first commit being used in the stdlib). And since we are not deprecating shutil here as that's not really benefiting the preexisting users, you have to have a really good reason for increasing the amount of code you're asking me to support for a couple of decades (on top of everything else I already have to support and will need to support in the future).

Now, I do understand that @barneygale has plans for an AbstractPath so we can have a universal API for path and file manipulation that can transcend file systems. Great! But that dream has not come true yet, so that isn't an argument (yet) to increase the API service. Once again, I have been around long enough to see good intentions not be completed to rely on it to make other decisions. This isn't a race; Python 3.12, 3.13, 3.14, etc. will still get released, so there's no rush to get anything in now. But yes, AbstractPath could change how we look at a lot of this.

@Ovsyanka83 what @merwok said covers it nicely; common actions can/should be on pathlib, but things that are lower-level or more rare should lean on __fspath__ . But there's also the consideration that duplicated APIs are twice the APIs to support since tossing out shutil for pathlib is a wide-ranging breaking change.