A flowchart demonstrating the exploit described.

Vibe Check: False Packages A New LLM Security Risk?

April 12, 2025 by Tyler August 23 Comments

Lots of people swear by large-language model (LLM) AIs for writing code. Lots of people swear at them. Still others may be planning to exploit their peculiarities, according to [Joe Spracklen] and other researchers at USTA. At least, the researchers have found a potential exploit in ‘vibe coding’.

Everyone who has used an LLM knows they have a propensity to “hallucinate”– that is, to go off the rails and create plausible-sounding gibberish. When you’re vibe coding, that gibberish is likely to make it into your program. Normally, that just means errors. If you are working in an environment that uses a package manager, however (like npm in Node.js, or PiPy in Python, CRAN in R-studio) that plausible-sounding nonsense code may end up calling for a fake package.

A clever attacker might be able to determine what sort of false packages the LLM is hallucinating, and inject them as a vector for malicious code. It’s more likely than you think– while CodeLlama was the worst offender, the most accurate model tested (ChatGPT4) still generated these false packages at a rate of over 5%. The researchers were able to come up with a number of mitigation strategies in their full paper, but this is a sobering reminder that an AI cannot take responsibility. Ultimately it is up to us, the programmers, to ensure the integrity and security of our code, and of the libraries we include in it.

We just had a rollicking discussion of vibe coding, which some of you seemed quite taken with. Others agreed that ChatGPT is the worst summer intern ever. Love it or hate it, it’s likely this won’t be the last time we hear of security concerns brought up by this new method of programming.

Special thanks to [Wolfgang Friedrich] for sending this into our tip line.

The White House Memory Safety Appeal Is A Security Red Herring

February 29, 2024 by Maya Posch 107 Comments

In the Holy Programming Language Wars, the lingua franca of system programming – also known as C – is often lambasted for being unsecure, error-prone, and plagued with more types of behavior that are undefined than ones that are defined by the C standards. Many programming languages were said to be ‘C killers’, yet C is still alive today. That didn’t stop the US White House’s Office of the National Cyber Director (ONCD) from putting out a report in which both C and C++ got lambasted for being ‘unsafe’ when it came to memory management.

The full report (PDF) is pretty light on technical details, while citing only blog posts by Microsoft and Google as its ‘expert sources’. The claim that memory safety issues are the primary cause of CVEs is not substantiated, or at least ignores the severity of CVEs when looking at the CISA statistics for active exploits. Beyond this call for ‘memory safety’, the report then goes on to effectively call for more testing and validation, while kicking in doors that were opened back in the 1970s already with the Steelman requirements and the High Order Language Working Group (HOLWG) of 1975.

What truly is the impact and factual basis of the ONCD report?

Continue reading “The White House Memory Safety Appeal Is A Security Red Herring” →

Overwriting A Protected AVR Bootloader

July 5, 2014 by Eric Evenchick 9 Comments

A bootloader is typically used to update application code on a microcontroller. It receives the new program from a host, writes it to flash, verifies the program is valid, and resets the microcontroller. Perhaps the most ubiquitous example is the Arduino bootloader which allows you to load code without an AVR programmer.

The bootloader resides in a special part of memory, which is protected. On the AVR, it isn’t possible to write to the bootloader memory from the application code. This is to prevent you from accidentally breaking the bootloader and bricking the device.

However, it can be useful to write to the bootloader memory. The best example would be when you need to update the bootloader itself. To accomplish this, [Julz] found a workaround that defeats the AVR bootloader protection.

The challenge was to find a way to execute the Store Program Memory (spm) instruction, which can only be executed by the bootloader. [Julz] managed to make use of the spm instruction in the existing bootloader by counting cycles and modifying registers at the right time.

Using this technique, which [Julz] calls BootJacker, the Fignition 8 bit computer could have its bootloader updated. However, this technique would likely allow you to modify most bootloaders on AVR devices.