Avoid unnecessarily loading the hackage index

[mgsloan]

Loading the hackage is expensive, because the encoded index cache is 18Mb. On my fairly fast computer, it takes 0.3 seconds to decode it. Search getPackageCaches to find the spots it's used.

There are two cases where loading the hackage currently happens unnecessarily:

1. In loadSourceMap, because the user might have specified a package target on the commandline which is not in the resolver. In this case, it uses the latest from hackage.

2. In constructPlan, because plan construction errors get annotated with "latest applicable" versions.

For both of these, loading the hackage index is only necessary under specific and abormal circumstances. However, currently this expensive operation is being done regardless of need.

It's also used in resolveBuildPlan, but that only seems to be used by the tests.

[luigy]

Loading the hackage is expensive, because the encoded index cache is 18Mb. On my fairly fast computer, it takes 0.3 seconds to decode it.

Hmm, eventually we got to steal hoogle decoding tricks ⚡, but in the meantime lazy me just wants to write a separate cache like I do in #1839 since it looks like there are a few cases in which all we care about is the latest version. I tested throwing this info in the same cache and the size is around ~505Kb.

2. In constructPlan, because plan construction errors get annotated with "latest applicable" versions.

The smaller cache mentioned above can help with this and also for show command in #1614

$ stack show latest-version base
4.8.2.0

Also as another enhancement we can fork reading the cache as soon as we hit a command that might need it + using an MVar for configPackageCaches

[mgsloan]

Hmm, eventually we got to steal hoogle decoding tricks , but in the meantime lazy me just wants to write a separate cache like I do in #1839 since it looks like there are a few cases in which all we care about is the latest version. I tested throwing this info in the same cache and the size is around ~505Kb.

Currently working on a fast serialization library - this is one of the applications we have in mind! https://www.fpcomplete.com/blog/2016/03/efficient-binary-serialization

Also as another enhancement we can fork reading the cache as soon as we hit a command that might need it + using an MVar for configPackageCaches

True! In general we might be able to make things a little faster by running such blocking fetches in parallel.

[luigy]

Currently working on a fast serialization library - this is one of the applications we have in mind!

Excellent!
so if I want this now, I have to be join the fp force... hmm 😜

[mgsloan]

No promises, but should be released soon particularly if it improves performance a bunch for the stack usecase.

[sjakobi]

I have realized that one of the most frequent uses of the package caches is for lookups where we want to get the existing versions of a given package. That's an operation that isn't well supported by the current representation of the package caches which uses PackageIdentifier as the key.

So I'd suggest changing that representation to Map PackageName (Map Version PackageCache) or even HashMap PackageName .... The downside of using a HashMap is that we'd apparently have to define an orphan instance for Binary.