Introduce standard sharded SCIP index format

[varungandhi-src]

Protobuf has a size limit of 2GB per message.

A single index for Chromium is about 6GB, triggering an error in scip-clang.

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/message_lite.cc:402] scip.Index exceeded maximum protobuf size of 2GB: 6138839817

The indexer needs to shard the data ahead-of-time and emit that instead.

Proposed sharded index format:

• Directory containing one or more *.shard.scip files, each of which contains a scip.Index. All shards must have the same metadata.
• The default name for the outer directory will be index.scip.

src-cli can be pointed to index.scip with -file (maybe we should rename this flag?). It will be responsible for compressing/tarring and uploading the index.

The behavior should be as-if:

• The metadata field of a scip.Index was populated by the metadata field in some shard.
• Any other fields of scip.Index (currently documents and external_symbols) in *.shard.scip files in the archive are processed in lexicographic ordering based on shard file names.

We should add documentation about this format in the README or in the scip.proto file.

This feature requires changes in:

• The backend to accept the new sharded format. https://github.com/sourcegraph/sourcegraph/pull/51132
• The lib/codeintel/upload package in the Sourcegraph monorepo, to create the tar archive. https://github.com/sourcegraph/sourcegraph/pull/51134
• src-cli, to use the newer lib/codeintel/upload package and pass the right content type header
• scip-clang: to emit large indexes in the sharded format
• This repo: docs update.

Maybe we should also mention this feature addition in various CHANGELOGs.

[olafurpg]

It's possible to stream one document at a time when reading and writing a Protobuf payload. We do this in scip-java by appending the binary payload of a scip.Index with a single document https://sourcegraph.com/github.com/sourcegraph/scip-java@b847cd1fcc5280bb078214920b13a8eff1d4fcc8/-/blob/scip-semanticdb/src/main/java/com/sourcegraph/scip_semanticdb/ScipSemanticdb.java?L186

You can similarly read one document at a time from index.scip files by decoding one scip.Index tag. IIRC, the Go bindings don't have any dedicated API to do this but I've managed to implement it by doing the tag decoding by hand (it's not so much code).

[varungandhi-src]

Thanks for pointing this out, that obviates the need for the sharded format, at least from the indexer's side.

Sharing some test code in case someone wants to verify the point made above:

package main

import (
	"fmt"
	"github.com/golang/protobuf/proto"
	"github.com/sourcegraph/scip/bindings/go/scip"
	"os"
)

func main() {
	var i1 scip.Index
	var m1 = scip.Metadata{}
	var t1 = scip.ToolInfo{}
	t1.Name = "Hello"
	m1.ToolInfo = &t1
	i1.Metadata = &m1
	var d1 scip.Document
	d1.RelativePath = "a/b/c"
	var d2 scip.Document
	d2.RelativePath = "c/d/e"
	i1.Documents = append(i1.Documents, &d1, &d2)
	b1, err := proto.Marshal(&i1)
	if err != nil {
		panic(fmt.Sprintf("error on first marshal: %v", err))
	}
	f1, err := os.OpenFile("tree.scip", os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
	if err != nil {
		panic(fmt.Sprintf("failed to open tree.scip: %v", err))
	}
	defer f1.Close()
	f1.Write(b1)
	{
		var j0 scip.Index
		j0.Metadata = &m1
		var j1 scip.Index
		j1.Documents = append(j1.Documents, &d1)
		var j2 scip.Index
		j2.Documents = append(j2.Documents, &d2)
		f2, err := os.OpenFile("stream.scip", os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
		if err != nil {
			panic(fmt.Sprintf("failed to open stream.scip: %v", err))
		}
		for _, j := range []*scip.Index{&j0, &j1, &j2} {
			b, err := proto.Marshal(j)
			if err != nil {
				panic(fmt.Sprintf("error on marshalling: %v", err))
			}
			f2.Write(b)
		}
	}
}

❯ sha256sum *.scip
30314fc029facf0067cf8debaee5305470c35d7bfc18be4fa564f3c40c54ba31  stream.scip
30314fc029facf0067cf8debaee5305470c35d7bfc18be4fa564f3c40c54ba31  tree.scip

Closing this issue and the associated PRs.