Commit esp8266/Arduino@9790e1c break git installation

[luc-github]

-## Basic Infos

• [ x] This issue complies with the issue POLICY doc.
• [ x] I have read the documentation at readthedocs and the issue is not addressed there.
• [ x] I have tested that the issue is present in current master branch (aka latest git).
• [x ] I have searched the issue tracker for a similar issue.
• [N/A ] If there is a stack dump, I have decoded it.
• [ x] I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [latest git ]
• Development Env: [Arduino IDE]
• Operating System: [Windows 10 python 2.7.13 and 2.7.15]

Settings in IDE N/A

Problem Description

Commit 9790e1c cause git install to fail
update python 2.7.13 to 2.7.15 does not solve the issue

mcve

C:\Users\User\Documents\Arduino\hardware\esp8266com>git clone https://github.com/esp8266/Arduino.git esp8266
Cloning into 'esp8266'...
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 21186 (delta 6), reused 14 (delta 0), pack-reused 21158R
Receiving objects: 100% (21186/21186), 71.20 MiB | 1.49 MiB/s, done.

Resolving deltas: 100% (12571/12571), done.

C:\Users\User\Documents\Arduino\hardware\esp8266com>cd esp8266

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>git submodule update --init
Submodule 'libraries/SoftwareSerial' (https://github.com/plerup/espsoftwareserial.git) registered for path 'libraries/SoftwareSerial'
Submodule 'tools/esptool' (https://github.com/espressif/esptool.git) registered for path 'tools/esptool'
Submodule 'tools/pyserial' (https://github.com/pyserial/pyserial.git) registered for path 'tools/pyserial'
Submodule 'lwip2' (https://github.com/d-a-v/esp82xx-nonos-linklayer.git) registered for path 'tools/sdk/lwip2/builder'
Submodule 'tools/sdk/ssl/bearssl' (https://github.com/earlephilhower/bearssl-esp8266) registered for path 'tools/sdk/ssl/bearssl'
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/libraries/SoftwareSerial'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/esptool'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/pyserial'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/sdk/lwip2/builder'...
Cloning into 'C:/Users/User/Documents/Arduino/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl'...
Submodule path 'libraries/SoftwareSerial': checked out '23ae000cb2cf4d5823a2744f6b8ae831575ff135'
Submodule path 'tools/esptool': checked out '9ad444a6e06e58833d5e6044c1d5f3eb3dd56023'
Submodule path 'tools/pyserial': checked out 'c54c81d933b847458d465cd77e96cd702ff2e7be'
Submodule path 'tools/sdk/lwip2/builder': checked out 'a501b5722bd050f41a9351e69ca75937150ec138'
Submodule path 'tools/sdk/ssl/bearssl': checked out '67786877341aac98c62e3b765fc64f4c49d81370'

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done
Extracting dist/python-3.7.2.post1-embed-win32.zip
Traceback (most recent call last):
  File "get.py", line 121, in <module>
    main()
  File "get.py", line 118, in main
    get_tool(tool)
  File "get.py", line 84, in get_tool
    unpack(local_path, '.')
  File "get.py", line 61, in unpack
    rename_to = re.match(r'^([a-z][^\-]*\-*)+', dirname).group(0).strip('-')
AttributeError: 'NoneType' object has no attribute 'group'

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

doing installation prior to commit 9790e1c works as expected

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>git checkout e7e7a4da177c8d511ba363bb09ccb50674cf2374
....
HEAD is now at e7e7a4da... prepare allman style (#5774)

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading i686-w64-mingw32.esptool-f80ae31.zip
Done
Extracting dist/i686-w64-mingw32.esptool-f80ae31.zip
Downloading i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Done
Extracting dist/i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Downloading i686-w64-mingw32.mkspiffs-7fefeac.zip
Done
Extracting dist/i686-w64-mingw32.mkspiffs-7fefeac.zip

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

[reaper7]

I have same problem:
my comment: 9790e1c#commitcomment-32366937

[d-a-v]

@reaper7 It's easier to follow issues when they are "issues".
If it persists after #5791, can you please open a new issue ?

[reaper7]

#5791 fixed problem with downloading necessary files via get.py
but some more problems still exists #5792

[luc-github]

I can confirm download is Ok :

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done
Extracting dist/python-3.7.2.post1-embed-win32.zip
Downloading i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Done
Extracting dist/i686-w64-mingw32.xtensa-lx106-elf-20ed2b9c.zip
Downloading i686-w64-mingw32.mkspiffs-7fefeac.zip
Done
Extracting dist/i686-w64-mingw32.mkspiffs-7fefeac.zip

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

But I got issue when compiling :

Compiling core...
"C:\\Users\\User\\Documents\\Arduino\\hardware\\esp8266com\\esp8266/tools/python/python.exe" "C:\\Users\\User\\Documents\\Arduino\\hardware\\esp8266com\\esp8266/tools/signing.py" --mode header --publickey "C:\\Users\\User\\Documents\\gitlab\\ESP3D\\esp3d/public.key" --out "C:\\Users\\User\\AppData\\Local\\Temp\\arduino_build_368303/core/Updater_Signing.h"
Using library ESP8266WiFi at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi 
Using library ESP8266WebServer at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WebServer 
Using library EEPROM at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\EEPROM 
Using library ESP8266SSDP at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266SSDP 
Using library ESP8266mDNS at version 1.2 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266mDNS 
Using library ArduinoOTA at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\ArduinoOTA 
Using library DNSServer at version 1.1.1 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\DNSServer 
Using library arduinoWebSockets at version 2.1.0 in folder: C:\Users\User\Documents\Arduino\libraries\arduinoWebSockets 
Using library Hash at version 1.0 in folder: C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\libraries\Hash 
exec: "C:\\Users\\User\\Documents\\Arduino\\hardware\\esp8266com\\esp8266/tools/python/python.exe": file does not exist
Error compiling for board Generic ESP8266 Module.

which is true as python.exe is in tools directory:

[MarkusAD]

exec: "C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/python/python.exe": file does not exist

path refers to tools/python/python.exe but python.exe is in tools (with no python subdir).

[luc-github]

Yes this is what I describe in screen shots : error is correct not python.exe in python directory as it is directly in tools

[luc-github]

Looks like @d-a-v @earlephilhower there was some update upstream but it break again sorry
I do a full delete and full git download and

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266>cd tools

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done200%
Hash mismatch for dist/python-3.7.2.post1-embed-win32.zip, delete the file and try again
Traceback (most recent call last):
  File "get.py", line 121, in <module>
    main()
  File "get.py", line 118, in main
    get_tool(tool)
  File "get.py", line 83, in get_tool
    raise RuntimeError()
RuntimeError

Delete the file in dir and redo

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>python get.py
Platform: i686-mingw32
Downloading python-3.7.2.post1-embed-win32.zip
Done200%
Hash mismatch for dist/python-3.7.2.post1-embed-win32.zip, delete the file and try again
Traceback (most recent call last):
  File "get.py", line 121, in <module>
    main()
  File "get.py", line 118, in main
    get_tool(tool)
  File "get.py", line 83, in get_tool
    raise RuntimeError()
RuntimeError

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266\tools>

[earlephilhower]

You need to apply the latest PR, it adjusts the SHA256 sig and the zip file to "v2". The old one doesn't work anyway, so the old file was removed from the repo to avoid other people ending up with messy tools dirs.

[mcspr]

regarding #5792 (comment) os.fork use, not to spread discussion across issues. this happened to be 🔥 hot topic while I was trying this... sigh

Trying out with subprocess.call([sys.executable, esptool_path, ...]) it does work, but with one quirk.

Because python is embedded, it does not accept PYTHONPATH environment var (i.e. sys.path). There is python/python37._pth that is used to load modules. By default it is like this:

python37.zip
.

So one can either add pyserial.zip line into this file and replace submodule with pyserial archive, or just symlink serial directory from tools/pyserial. Both should work on Windows.

[luc-github]

@earlephilhower I did not saw PR was not merged - will apply and report

[earlephilhower]

@mcspr that's a good idea. I think a cleaner one would be, since I have to build a new ZIP file for python because the -embed one does not have a subdir: Just add a pyserial installation to the python-embed.zip itself, dropping pyserial as a tool installation/git repo completely.

I had hoped to just take the python ZIP as-is, but that's obviously not possible now that I see it doesn't actually include a subdir and that makes Arduino go sideways.

[luc-github]

@earlephilhower I confirm :

• PR fix installation issue on my side / Zip modification
• Zip modification upstream fix also compilation issue with python path
• problem of upload as described above is present

C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/python/python.exe C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/upload.py C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/pyserial C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/esptool/esptool.py --chip esp8266 --port COM7 --baud 115200 --trace version --end --chip esp8266 --port COM7 --baud 115200 --trace write_flash 0x0 C:\Users\User\AppData\Local\Temp\arduino_build_189745/esp3d.ino.bin --end 
Traceback (most recent call last):
  File "C:\Users\User\Documents\Arduino\hardware\esp8266com\esp8266/tools/upload.py", line 24, in <module>
    pid  = os.fork()
AttributeError: module 'os' has no attribute 'fork'
AttributeError: module 'os' has no attribute 'fork'