Remove SuiteSparse and derived functions from julia repo

[andreasnoack]

Instead, it becomes a proper package. The new package is https://github.com/JuliaLinearAlgebra/SuiteSparse.jl. This is similar to #27616.

[ViralBShah]

Putting on milestone to make sure we do this before releasing 0.7 or hopefully even before the beta.

[StefanKarpinski]

So if this is not ready this week, we should hold the beta until it's ready?

[ViralBShah]

Maybe review at end of week. If this is going to take substantially longer, we can move it off the milestone. I suspect we can get it in.

[ViralBShah]

Ok, the failing tests are because of norm. stdlib/SparseArrays/src/linalg.jl probably has to be moved out into its own package, or into SuiteSparse - so that Base only has SparseArrays. These things were too tightly coupled to begin in the first place.

The second option is to move all sparse capabilities out of stdlib.

The third option is to not move out SuiteSparse at all.

As I experiment with this, I think we should either have sparse matrices and sparse linear algebra both in stdlib, or move all of sparse out of stdlib. Keeping it piecemeal could be confusing.

@Sacha0 @KristofferC @andreasnoack @fredrikekre

[andreasnoack]

The reason why this isn't completed yet is exactly the sparse linear algebra components in SparseArrays. I've started to factor them out into a separate SparseLinearAlgebra package but this happens to exercise some corner cases in the package manager as well but we are getting there.

[andreasnoack]

Tests pass locally now. I've moved the linear algebra pieces that depend on SuiteSparse to https://github.com/JuliaLinearAlgebra/SparseLinearAlgebra.jl which will be registered shortly. I'd need to add a few deprecations here but besides that, I think it's ready. There are some linear algebra routines left in SparseArrays that would be nice to migrate to SparseLinearAlgebra but they don't depend on SuiteSparse so it doesn't have to happen as part of this PR.

[ViralBShah]

I think we should just move all of the sparse linear algebra routines in one shot, rather than do some here and some later, so that there is a clear link between what was removed and what is available in SparseLinearAlgebra.

[andreasnoack]

I've moved the remaining linear algebra pieces to SparseLinearAlgebra. The separation between linear algebra and not linear algebra isn't always clear. I've decided to keep some of the kernel functions such as e.g. fkeep and permute! in SparseArrays and only move the linear algebra consumers of these such as tril and transpose to the new package.

Another concern is the effect of the generic linear algebra fallbacks. Moving the sparse linear algebra functions to SparseLinearAlgebra shouldn't break much because of the generic fallbacks but it will cause huge slowdowns unless users load SparseLinearAlgebra. We could make an announcement on Discourse to make clear that most users in the future should prefer SparseLinearAlgebra over SparseArrays.

[ViralBShah]

Having the default implementation that works but is slow is not ideal. Can we disable the fallback for SparseMatrixCSC?

[andreasnoack]

Can we disable the fallback for SparseMatrixCSC?

I don't think this is practical. The two proper solutions would be to either change the signatures of all the fallbacks since there are many arrays for which they're not really useful (SparseMatrixCSC being one of them and DArray another) or to say the fallbacks got it right and that the problem is that SparseMatrixCSC shouldn't be an AbstractMatrix. I think neither is feasible at this point. Adding linear algebra methods that will just error in SparseArrays doesn't seem like a good solution so I think we'll have to live the with the risk of slow fallbacks. In a better future, we might be able to solve this issue by utilizing dispatch on memory layout but we are not there yet.

[ararslan]

There are still some references to spdiagm, which is causing the test failures.

[ViralBShah]

What about the option of moving all of the sparse matrix capabilities into an external package? It seems that it is better to provide a reliable way to get good performance rather than accidentally leading people to slow fallbacks - we will keep receiving messages about the slowness and have to keep repeating the solution.

[ViralBShah]

Let's get this PR merged and move SuiteSparse out first. The other questions that result from this, we can deal with as we move forward.

[andreasnoack]

What about the option of moving all of the sparse matrix capabilities into an external package?

Making all of SparseArrays a proper package is indeed an alternative and would be much simpler than what I've been doing here. The price of moving out all of SparseArrays instead of this PR would be that we wouldn't get the decoupling of the spares data structures and linear algebra operations on them. You could argue, though, that the fallbacks make a complete decoupling impossible. I still think the decoupling that this PR provides is worthwhile since it will give us some flexibility with respect to which direct solvers we pull in if any at all.

[ViralBShah]

Yes, decoupling sparse solvers does give tremendous flexibility. Let's do that first and think about SparseArrays as we go along.

[StefanKarpinski]

I’m very against moving this now that we have got the beta cut. Deadlines have to mean something and we need to make progress toward a final release and stop pushing large breaking changes to the last minute.

[ararslan]

@andreasnoack I rebased the branch to fix the remaining merge conflict, so this should be good once CI passes.

[ararslan]

More conflicts were introduced, so I've rebased again.

[ViralBShah]

Given that this moves a bunch of matmul stuff as well, quite a bit of stuff would slow down due to fallbacks. I would personally be favour of either retaining SuiteSparse, or moving all of sparse (including SparseArrays), but not do it piecemeal.

[KristofferC]

I agree with @ViralBShah. I don't think it is worth to move things out until we can point to a SparseLinearAlgebra meta package.

[andreasnoack]

I don't think it is worth to move things out until we can point to a SparseLinearAlgebra meta package.

https://github.com/JuliaLinearAlgebra/SparseLinearAlgebra.jl is already there and can be finished up and registered if we want to merge this PR so that is not the concern. @ViralBShah's concern is if people will remember to write using SparseLinearAlgebra instead of using SparseArrays when some of the linear algebra already works (though really slowly through the generic fallbacks) even when only the latter is loaded.

For Arrays, we have a module separation between the data structure and the linear algebra operations so I think it makes sense for sparse linear algebra as well but the issue is that the generic fallbacks work for sparse arrays.

[KristofferC]

With type piracy, there will always be that concern. Perhaps it can be solved by being clear in the README's of the package what the purpose of them are?

[ViralBShah]

With dense arrays we have the separation, but from a user standpoint, arrays are in Base, and you only have to import LinearAlgebra. People will not know or remember the distinction between SparseArrays and SparseLinearAlgebra. In fact, even I don't think I will.

Maybe we can have a set of specializations that basically say "use SparseLinearAlgebra" - should be easy with a macro. With that, I wouldn't mind splitting this out. Shipping slow behaviour by default is something we should avoid.

[andreasnoack]

The idea is that most people shouldn't need to remember the distinction between SparseArrays and SparseLinearAlgebra. Most user should just know SparseLinearAlgebra but those few who really want to avoid the sparse linear algebra and just use the data structure would be able to do so.

[ViralBShah]

How will people learn about SparseLinearAlgebra? Imagine a new user who downloads Julia and starts using it and runs into slow stuff. Then do you have to go to stackoverflow and figure this out?

[fredrikekre]

It is unlikely that you get your hands on a SparseMatrixCSC without loading either SparseArrays or SparseLinearAlgebra, we just have to promote SparseLinearAlgebra over SparseArrays.

[ararslan]

Can we add a dependency on LinearAlgebra to SparseArrays and define the methods that would otherwise hit slow fallbacks to throw an error pointing to SparseLinearAlgebra?

[ViralBShah]

How do we promote the one thing that is not in the distribution over one thing that is? Should our Julia docs then ask people to install the external package and maybe not ever refer to SparseArrays? Technically yes, but as a user I would find this extremely confusing.

[ararslan]

So can we move SparseArrays out as well?

[ViralBShah]

Probably needs to be redone differently whenever we reconsider this.