py command fails with RC_INTERNAL_ERROR (109) return code when run with an App Pool identity in Windows

[kenlyon]

Bug report

Bug description:

I have a .NET 6 web application that responds to REST requests. Amongst other things, it can execute python scripts. This is achieved by starting a new process that uses the "py" command to execute our script.

I recently discovered that this is failing with an exit code of 109, which is described in the documentation as "Unexpected error. Please report a bug."

I had installed python 3.12.1 for all users on the host machine. I also found that when I'm debugging the application (it runs as me) it works fine, but when it's installed in IIS (running as the ApplicationPoolIdentity) it always fails with that error, even if I try to just run "py -h".

I did find that if I changed my Application Pool Identity to be "LocalSystem" it did work.

I also found that if I changed my code to execute "C:\Program Files\Python312\python.exe" it worked correctly.

It seems to be a problem with py.exe trying to discover the installed versions of python while being executed with an AppPoolIdentity.

Is this a bug in py.exe? Is there some special permission I need to grant my AppPoolIdentity on a particular folder or something?

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-115049: Fix py.exe failing when user has no LocalAppData. #115185
• [3.12] gh-115049: Fix py.exe failing when user has no LocalAppData. (GH-115185) #115354

[zooba]

Looking at our sources, it seems possible that SHGetFolderPathW is failing. We have one place where this function failing could result in that exit code without an error message - virtually every other place it can happen will display error text.

I'm not sure exactly what circumstances this will fail under, but it seems possible that the IIS user may not have an actual profile directory, which could cause it. Running as an actual user means there'll be a profile directory and it will succeed.

We shouldn't be failing "hard" at this point. If we can't find the directory, then the user can't possibly have any installs in it, so we can just skip it. That's something we can fix in an update.

In the meantime, is there a reason you aren't able to provide the full executable path to your app? Maybe as a configuration setting via IIS? It seems... brave to have a web service guess where it'll launch Python from each time.

[kenlyon]

Thanks for your quick response. That makes sense that it could be getting confused looking up folders belonging to the IIS user. I had a quick look in C:\Users and it does have a folder there but perhaps the resolution of the path is getting messed up in some way.

There are a few reasons we don't provide the full path to python. For one, our customers may have certain other software installed side-by-side with ours which includes python. In that situation we want to use their version. In other cases, it might be running on linux, so we do something else. Finally for Windows we hoped that using "py.exe" would automatically find the relevant version, doing the work for us.

Maybe we need to inspect HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore for ourselves and discover what's there.

[zooba]

The only things that will be in that registry key are things you've installed yourself :) (or an attacker/intern has installed for you... that's the "risk" inherent in dynamically choosing how to run your app)

py.exe can only know the "relevant" version if you tell it which version to use (e.g. py.exe -V:PythonCore/3.13 ...). So if a particular customer wants a different version to run, they'd probably need to update something anyway.

I assume this wouldn't be the only customisable setting you have for customers? Maybe you have branding/company name information or login servers saved somewhere for them? That might be a suitable place to put C:\Program Files\Python312\python.exe and then call it directly.

[kenlyon]

The only things that will be in that registry key are things you've installed yourself :) (or an attacker/intern has installed for you... that's the "risk" inherent in dynamically choosing how to run your app)

Again, this isn't quite how it works. Our product is installed by customers on their own servers. We trust them to be in control of what else they have installed on there. If the customer has installed python, we need to determine how to run it.

It's true that we could require the customer to specify the location of python.exe during the installation process. We do that for certain things like email and database configuration. However, the hope was that we could discover it without needing to give them something else to configure.

It was my understanding that the py.exe command could be used as a sort of alias, allowing that discovery process to be abstracted away from us. As stated in the initial report, this does work as long as the user isn't an app pool user.

py.exe can only know the "relevant" version if you tell it which version to use (e.g. py.exe -V:PythonCore/3.13 ...). So if a particular customer wants a different version to run, they'd probably need to update something anyway.

If they wanted to use a different version, wouldn't it be possible to use the special style of comment at the top of the file to indicate the version the script needs?

[kenlyon]

I tested some old versions from the Windows Downloads page and found that the first version with this problem was 3.11.0a7. It had worked in 3.11.0a6.

Hopefully that helps find the breaking change.

[kenlyon]

Maybe it was this change?
#90724

That fits the timeline and might be the combination of SHGetFolderPathW and RC_INTERNAL_ERROR that you referred to in your initial response.

[zooba]

Thanks for confirming that's probably the issue, it's what I suspected (I have other changes ongoing right now, so haven't been able to do anything on this one).

We generally intend py.exe for interactive use, an alternative to searching the PATH environment variable (which can become inconsistent very easily). But in your case it may well be the best option, especially if you do want to allow users to specify shebang lines.

FWIW, there should be no licensing issues if you want to include your own copy of an older py.exe until we get a fix done. It has no dependencies and can run from anywhere. Older versions may still have some bugs relating to shebang lines and environment variables, so choose your problems I guess. (It's also only a single .c file in our repo, so if you wanted to compile it from source yourself that would also be fine.)

[kenlyon]

Thanks for the suggestion. I'm going to go with your earlier idea of additional configuration. Our customers will be able to specify the path to python.exe via a configuration file. As long as they installed it for all users, it will be fine. We'll still support py.exe for those who wish to use it, with the caveat that there's been this issue since v3.11.

[zooba]

You could make that configuration default to py.exe if you wanted, which would preserve your existing behaviour. We guarantee that all python.exe command lines will work with py.exe.

[kenlyon]

Yes, that's the plan. In the absence of the path being specified in the new configuration, the product will fall back to the previous logic.