Flashing an Arduino Leonardo with AVRDUDE is troublesome

[kristofmulier]

The ATmega32u4 microcontroller is present on a couple of Arduino boards, such as the Arduino Leonardo and Arduino Micro. The flash procedure for these boards is quite complicated. I'll first review how it works precisely, then I'll explain why this is a problem for many users of AVRDUDE. Finally we can look into potential solutions.

1. The flash procedure explained

Unlike others, the Arduino Leonardo and Arduino Micro boards identify themselves through one port (eg. COM9) while you need to flash them through another (eg. COM10) which is only available for a couple of seconds just after a reset. I observed how the Arduino IDE deals with this case:

1. Open the serial port through which the board has identified itself (eg. COM9) at baudrate 1200 and close it immediately thereafter. This basically triggers a board reset.

2. Observe the serial ports from the OS (Windows/Linux). As soon as the board has completed the reset, it should present itself with the serial port for flashing (eg. COM10). Store that port name and go on to the next step.

3. Flash the firmware through the new port (eg. COM10) with AVRDUDE. Do it quickly, because this serial port is only active for a couple of seconds.

Arduino IDE does it this way. PlatformIO also.

 

2. What is the problem?

The flash procedure explained above consists of three steps. I know how to put the third step in a makefile, but I have no idea how to implement the first two.

I could just hardcode the first two steps in Embeetle IDE and run them behind the scenes before launching the third step. I actually just did that, and it works. But it feels wrong. At Embeetle, we have the desire to keep our users informed and free. In other words: if one of our users decides to work on a project outside Embeetle IDE, he/she is free to do so. We achieve that in two ways:

• All Embeetle projects are makefile-based. The user simply needs to invoke clean, build and flash targets to do all the magic.

• We provide a makefile to each sample project our users can download, but it is not touched thereafter. The makefile is well-explained, with plenty of comments.

Hardcoding (part of) the flash procedure in the IDE itself defeats this goal. The user cannot run his project outside our IDE anymore.

 

3. Potential solutions

I see two solutions to this dilemma:

1. We find a way to implement the first two steps in the makefile anyhow. I looked into this with my colleague Johan - who is a true makefile wizard - but we couldn't figure it out. If you have an idea, please let me know.

2. We improve the AVRDUDE software such that it can handle this particular situation.

I believe the second solution is the cleanest and will help most AVRDUDE users - also those that have nothing to do with Embeetle IDE. I would love to help with the implementation, but I'm afraid I need a little help and guidance, as I'm not familiar with the source code.

Kind regards,
Kristof Mulier

[dl8dtl]

The way the second device comes up is very OS-specific. I'd like to avoid that kind of OS hacks in AVRDUDE itself, as it opens a can of worms. So far, AVRDUDE suffers only from very few (low-level) things that are really OS specific, as for most of the required abstractions, it can sit on top of the respective backend libraries (libusb, hidapi and such).
Ideally, there ought to be a kind of OS service you can attach to that tells you when a new device arrives. You can then write a script that does:

• register at that OS service
• issue a reset
• wait until the new device came up though that OS service
• start AVRDUDE on it

Obviously, that script is inherently OS specific.

[kristofmulier]

Hi @dl8dtl ,

You basically suggest to defer the first two steps (1. resetting device and 2. waiting for it to return with a new serial port) to an OS-specific script outside AVRDUDE. A good idea, as this keeps AVRDUDE clean from OS-specific stuff.

A few questions come to my mind:

• In what language should the script be written?
  Personally, I think it should be a script that just runs on the user's computer without requiring anything in particular to be installed. Therefore, I would not do it in Python. Perhaps a shell script? I suppose it's doable in a Linux shell script, but quite horrific to achieve in a Windows .bat file. What about doing it in a small, separate little C program?

• How will this script be distributed with AVRDUDE?
  The typical AVRDUDE installation on Linux is very clean right now - just two files:

  📁 avrdude_6.3.0
  ├─ 📁 bin
  │  └─ avrdude
  └─ 📁 etc
     └─ avrdude.conf
  

  Maybe the script should be added in the bin/ folder. Or an extra folder scripts/ should be introduced?

• How will the script be invoked?
  I suppose AVRDUDE should invoke the script in the first place. Eventually, the script invokes AVRDUDE again in the end (see step 3 in the flash procedure). This way, the user doesn't even realize the first two steps are being deferred to a script outside AVRDUDE. Looks like the best user-experience to me.

Kind regards,
Kristof

[mariusgreuel]

Hi Kristof,

while I agree that it would be a nice addition to AVRDUDE, it is not quite trivial to do this cross-platform. I did this for Windows, check out
https://github.com/mariusgreuel/avrdude

The code for doing the auto-reset is here, it is just about 1k lines of non-portable code:
https://github.com/mariusgreuel/avrdude/blob/windows/src/msvc/usb_com_locator.h

[kristofmulier]

Hi @mariusgreuel ,
It's awesome you already did it for Windows! That means most of the work is already done. Usually such things are much harder to achieve on Windows than on Linux.

[dl8dtl]

it is just about 1k lines of non-portable code

Puh. I'm not sure whether I'd like to have that in a supposed-to-be portable application.

[MCUdude]

As a compromise, would it be an idea to create a separate "1200 bps touch" handler that would contain all the non-portable code one could call along with avrdude?

The boards I know of that are currently "impossible" to flash directly using Avrdude because of this is the Arduino Leonardo, Arduino Micro/Pro Micro, and the Arduino Nano Every.

I believe making this work on Windows is the most difficult part, but @mariusgreuel has already figured out this part. At least on my mac, the serial port always has the same name after a disconnect. The only time the name changes is when I use a different USB port.

The command below is just to illustrate what I mean. The idea1200bps_touch returns the serial port that "appears" after disconnecting at 1200 baud, and that the output is passed to avrdude as an argument. Clearly not that elegant, but it would be nice to have something we can utilize (except for uploading through Arduino IDE) when working with a board that requires this logic to work.

1200bps_touch -P /dev/cu.usbmodem14101 | xargs avrdude -Cavrdude.conf -cjtag2updi -patmega4809 -b115200

Here's the python based logic PlatformIO uses:

        before_ports = get_serial_ports()

        if upload_options.get("use_1200bps_touch", False):
            env.TouchSerialPort("$UPLOAD_PORT", 1200)

        if upload_options.get("wait_for_upload_port", False):
            env.Replace(UPLOAD_PORT=env.WaitForNewSerialPort(before_ports))

def TouchSerialPort(env, port, baudrate):
    port = env.subst(port)
    print("Forcing reset using %dbps open/close on port %s" % (baudrate, port))
    try:
        s = Serial(port=port, baudrate=baudrate)
        s.setDTR(False)
        s.close()
    except:  # pylint: disable=bare-except
        pass
    sleep(0.4)  # DO NOT REMOVE THAT (required by SAM-BA based boards)


def WaitForNewSerialPort(env, before):
    print("Waiting for the new upload port...")
    prev_port = env.subst("$UPLOAD_PORT")
    new_port = None
    elapsed = 0
    before = [p["port"] for p in before]
    while elapsed < 5 and new_port is None:
        now = [p["port"] for p in util.get_serial_ports()]
        for p in now:
            if p not in before:
                new_port = p
                break
        before = now
        sleep(0.25)
        elapsed += 0.25

    if not new_port:
        for p in now:
            if prev_port == p:
                new_port = p
                break

    try:
        s = Serial(new_port)
        s.close()
    except SerialException:
        sleep(1)

    if not new_port:
        sys.stderr.write(
            "Error: Couldn't find a board on the selected port. "
            "Check that you have the correct port selected. "
            "If it is correct, try pressing the board's reset "
            "button after initiating the upload.\n"
        )
        env.Exit(1)

    return new_port

[mcuee]

The boards I know of that are currently "impossible" to flash directly using Avrdude because of this is the Arduino Leonardo, Arduino Micro/Pro Micro, and the Arduino Nano Every.

I have the Arduino Leonardo and I am getting the Arduino Pro Micro and Arduino Nano Every soon. So this is interesting.

[mcuee]

This is what Arduino is doing: run log under Windows with a clone Pro Micro with ATmega32U4.

Forcing reset using 1200bps open/close on port COM18
PORTS {COM18, } / {} => {}
PORTS {} / {} => {}
PORTS {} / {COM8, } => {COM8, }
Found upload port: COM8
C:\Users\xiaof\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino17/bin/avrdude -CC:\Users\xiaof\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino17/etc/avrdude.conf -v -patmega32u4 -cavr109 -PCOM8 -b57600 -D -Uflash:w:C:\Users\xiaof\AppData\Local\Temp\arduino_build_628741/Blink_Every.ino.hex:i 

avrdude: Version 6.3-20190619
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "C:\Users\xiaof\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino17/etc/avrdude.conf"

         Using Port                    : COM8
         Using Programmer              : avr109
         Overriding Baud Rate          : 57600
         AVR Part                      : ATmega32U4
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  9000  9000 0x00 0x00
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0x00 0x00
           lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : butterfly
         Description     : Atmel AppNote AVR109 Boot Loader

Connecting to programmer: .
Found programmer: Id = "CATERIN"; type = S
    Software Version = 1.0; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=128 bytes.

Programmer supports the following devices:
    Device code: 0x44

avrdude: devcode selected: 0x44
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% -0.00s

avrdude: Device signature = 0x1e9587 (probably m32u4)
avrdude: reading input file "C:\Users\xiaof\AppData\Local\Temp\arduino_build_628741/Blink_Every.ino.hex"
avrdude: writing flash (3956 bytes):

Writing | ################################################## | 100% 0.31s

avrdude: 3956 bytes of flash written
avrdude: verifying flash memory against C:\Users\xiaof\AppData\Local\Temp\arduino_build_628741/Blink_Every.ino.hex:
avrdude: load data flash data from input file C:\Users\xiaof\AppData\Local\Temp\arduino_build_628741/Blink_Every.ino.hex:
avrdude: input file C:\Users\xiaof\AppData\Local\Temp\arduino_build_628741/Blink_Every.ino.hex contains 3956 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.05s

avrdude: verifying ...
avrdude: 3956 bytes of flash verified

avrdude done.  Thank you.

[mcuee]

Ref: https://www.avrfreaks.net/forum/uploading-code-new-arduino-nano-every
Trick under Windows using a batch file to use with Arduino Nano Every.

mode %COMPORT% baud=12 dtr=on > nul
mode %COMPORT% baud=12 dtr=off > nul

More details here:

• avrdude with Arduino Nano Every #980

[mcuee]

@mariusgreuel I tried your branch ( https://github.com/mariusgreuel/avrdude) and it seems to work for me for the Pro Micro clone.

Run log with a Pro Micro clone.

PS C:\work\avr\avrdude_test\avrdude_win\out\build\x64-Debug\src\Debug> .\avrdude.exe -patmega32u4 -c avr109 -PCOM14 -v

avrdude.exe: Version 7.0
             Copyright (c) Brian Dean, http://www.bdmicro.com/
             Copyright (c) Joerg Wunsch

             System wide configuration file is "C:/work/avr/avrdude_test/avrdude_win/out/build/x64-Debug/src/Debug/avrdude.conf"

             Using Port                    : COM14
             Using Programmer              : avr109
avrdude.exe: Found device at port 'COM14'
avrdude.exe: Entering bootloader mode...
avrdude.exe: Found device at port 'COM10'
             AVR Part                      : ATmega32U4
             Chip Erase delay              : 9000 us
             PAGEL                         : PD7
             BS2                           : PA0
             RESET disposition             : dedicated
             RETRY pulse                   : SCK
             Serial program mode           : yes
             Parallel program mode         : yes
             Timeout                       : 200
             StabDelay                     : 100
             CmdexeDelay                   : 25
             SyncLoops                     : 32
             PollIndex                     : 3
             PollValue                     : 0x53
             Memory Detail                 :

                                               Block Poll               Page                       Polled
               Memory Type Alias    Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
               ----------- -------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
               eeprom                 65    20     4    0 no       1024    4      0  9000  9000 0x00 0x00
               flash                  65     6   128    0 yes     32768  128    256  4500  4500 0x00 0x00
               lfuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
               hfuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
               efuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
               lock                    0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
               calibration             0     0     0    0 no          1    1      0     0     0 0x00 0x00
               signature               0     0     0    0 no          3    1      0     0     0 0x00 0x00

             Programmer Type : butterfly
             Description     : Atmel AppNote AVR109 Boot Loader

Connecting to programmer: .
Found programmer: Id = "CATERIN"; type = S
    Software Version = 1.0; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=128 bytes.

Programmer supports the following devices:
    Device code: 0x44

avrdude.exe: devcode selected: 0x44
avrdude.exe: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude.exe: Device signature = 0x1e9587 (probably m32u4)

avrdude.exe done.  Thank you.

[mcuee]

@mariusgreuel I tried your branch ( https://github.com/mariusgreuel/avrdude) under Windows but somehow it does not work for my official Arduino Leonardo as it stuck there forever.

PS C:\work\avr\avrdude_test\avrdude_win\out\build\x64-Debug\src\Debug> .\avrdude.exe -patmega32u4 -c avr109 -PCOM14 -vvvv
executable_abspath = C:/work/avr/avrdude_test/avrdude_win/out/build/x64-Debug/src/Debug/avrdude.exe
executable_abspath_len = 78
executable_dirpath = C:/work/avr/avrdude_test/avrdude_win/out/build/x64-Debug/src/Debug
executable_dirpath_len = 66
sys_config = C:/work/avr/avrdude_test/avrdude_win/out/build/x64-Debug/src/Debug/avrdude.conf
sys_config_found = true


avrdude.exe: Version 7.0
             Copyright (c) Brian Dean, http://www.bdmicro.com/
             Copyright (c) Joerg Wunsch

             System wide configuration file is "C:/work/avr/avrdude_test/avrdude_win/out/build/x64-Debug/src/Debug/avrdude.conf"

             Using Port                    : COM14
             Using Programmer              : avr109
avrdude.exe: Found device at port 'COM14'
avrdude.exe: - InstanceId: USB\VID_2341&PID_8036&MI_00\6&22A06EED&0&0000
avrdude.exe: - Port: COM14
avrdude.exe: - Location: Port_#0001.Hub_#0002
avrdude.exe: - ProductName: Arduino Leonardo
avrdude.exe: - IsPresent: true
avrdude.exe: - IsCompositeDevice: true
avrdude.exe: Entering bootloader mode... 
(stuck here even after WIndows changes to COM15)

It does work under Arduino.

Forcing reset using 1200bps open/close on port COM14
PORTS {COM14, } / {} => {}
PORTS {} / {COM15, } => {COM15, }
Found upload port: COM15
C:\Users\xiaof\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino17/bin/avrdude -CC:\Users\xiaof\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino17/etc/avrdude.conf -v -patmega32u4 -cavr109 -PCOM15 -b57600 -D -Uflash:w:C:\Users\xiaof\AppData\Local\Temp\arduino_build_926279/Blink_Every.ino.hex:i 

avrdude: Version 6.3-20190619
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "C:\Users\xiaof\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino17/etc/avrdude.conf"

         Using Port                    : COM15
         Using Programmer              : avr109
         Overriding Baud Rate          : 57600
         AVR Part                      : ATmega32U4
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  9000  9000 0x00 0x00
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0x00 0x00
           lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : butterfly
         Description     : Atmel AppNote AVR109 Boot Loader

Connecting to programmer: .
Found programmer: Id = "CATERIN"; type = S
    Software Version = 1.0; No Hardware Version given.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize=128 bytes.

Programmer supports the following devices:
    Device code: 0x44

avrdude: devcode selected: 0x44
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% -0.00s

avrdude: Device signature = 0x1e9587 (probably m32u4)
avrdude: reading input file "C:\Users\xiaof\AppData\Local\Temp\arduino_build_926279/Blink_Every.ino.hex"
avrdude: writing flash (3956 bytes):

Writing | ################################################## | 100% 0.31s

avrdude: 3956 bytes of flash written
avrdude: verifying flash memory against C:\Users\xiaof\AppData\Local\Temp\arduino_build_926279/Blink_Every.ino.hex:
avrdude: load data flash data from input file C:\Users\xiaof\AppData\Local\Temp\arduino_build_926279/Blink_Every.ino.hex:
avrdude: input file C:\Users\xiaof\AppData\Local\Temp\arduino_build_926279/Blink_Every.ino.hex contains 3956 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.04s

avrdude: verifying ...
avrdude: 3956 bytes of flash verified

avrdude done.  Thank you.