Enormous dependency tree — consider "drivers"?

[brandur]

Feature Request / Improvement

Given an empty project with Iceberg-go as its only dependency:

cat main.go

package main

import (
        _ "github.com/apache/iceberg-go"
        _ "github.com/apache/iceberg-go/catalog"
        _ "github.com/apache/iceberg-go/catalog/rest"
        _ "github.com/apache/iceberg-go/io"
        _ "github.com/apache/iceberg-go/table"
)

func main() {
}

head go.mod

module github.com/brandur/iceberg-go-test

go 1.25.6

require github.com/apache/iceberg-go v0.4.1-0.20260123180146-3ce3b4bb51c0

require (
        atomicgo.dev/cursor v0.2.0 // indirect
        atomicgo.dev/keyboard v0.2.9 // indirect
        atomicgo.dev/schedule v0.1.0 // indirect

Iceberg-go adds ~1s of compile overhead (on my commodity laptop). This might not be the end of the world, but you end up soaking it up on every test run compile and test run, which is fairly painful for any projects using Iceberg-go:

$ go test .
ok      github.com/brandur/iceberg-go-test      0.781s [no tests to run]

The project has about 130 dependencies. Manually inspecting the list, a big number of them are from the three major cloud providers (AWS, Azure, Google). Being able to jettison those would probably help a lot.

An approach that we took in one of our projects (River) is to use a "driver" system wherein you have sub-modules for major dependencies so users can pull them in piece by piece. e.g.

https://github.com/riverqueue/river/tree/master/riverdriver

It's far from perfect as Go's multi-module handling leaves a lot to be desired, but it works reasonably well, and a similar approach would probably help Iceberg-go in the longer run. Thoughts?

[zeroshade]

This is actually something similar that I was discussing with a few others that were having an issue with the gocloud.dev dependency.

Currently for the catalogs we have a registration config that allows for the go compiler to leave out catalog implementations that aren't being used. I've been considering doing something similar for the io management too. We've already isolated everything to the interfaces so it shouldn't be difficult to do a similar registration pattern for handling the io location handling.

Would you be willing to contribute something to this extent (or something like what you're suggesting)?

[brandur]

Would you be willing to contribute something to this extent (or something like what you're suggesting)?

Normally I'd be a little reluctant to try a fairly large change like this as a first time contributor as there'd be a lot of points of potential disagreement, but I just took a look at the iceberg/io code, and it seems to be quite constrained, so this should be pretty doable. Let me get a rough version together and pitch it up to see if we can get approximate agreement on what the interface for such a thing should look like.

Also, a heads up in case we go this route, but multiple modules will make things like the release process a little heavier weight. For example, we'll need to generate Git tags for all the submodules along with the top level directory. It can also make things like running tests a tad more difficult without dipping into features like Go workspaces, which are a fount of problems in their own way.

[zeroshade]

For an initial pass, let's start by a similar approach to how the catalog package works and avoid the multiple modules and just isolate packages and subpackages. we can look into potentially updating to manage the multiple modules in a second pass