Recoverable Error Handling

[martindevans]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

Use a form of error handling which doesn't immediately terminate the process (GGML_ASSERT).

Motivation

Currently llama.cpp checks various conditions with GGML_ASSERT and if it fails it will immediately abort(). There's no way to handle this error and it immediately kills the process, which makes it difficult to build robust services using llama.cpp. see for example: SciSharp/LLamaSharp#343 (comment)

This has presented an issue in LLamaSharp (which I'm one of the developers of). In C# it's more reasonable that an error might throw an exception, which can be handled. Unfortunately there's no way at all for us to implement this.

Possible Implementation

I don't know C++ well enough to comment. But from the perspective of LLamaSharp (and probably other wrappers) any kind of error signalling that didn't immediately terminate the process would be great!

[slaren]

We could do better in this respect, but the "CUDA error 700" in the linked issue indicates a programming error and something like that is probably never going to be recoverable. That specific error should have already been fixed.

[martindevans]

That particular issue was just the motivation for a larger change in error handling, thanks for confirming that though 👍

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[martindevans]

No activity, but hopefully still relevant as a long term goal!

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.