List and download dependencies to be generated before build? – getting help

Greetings! I have a potentially curious issue with pre-downloading dependencies for network-isolated container builds.

What I am trying to do:

  • via pre-download dependencies go mod download in a local directory
  • Mount the directory in container build, set GOMODCACHE to point to this directory
  • Build container without internet access

Problem: Part of the build process is running go generate, This command wants to download additional modules that were not downloaded by Go Mod Downloads. It fails because there is no internet access.

go: downloading golang.org/x/tools v0.1.13-0.20220804200503-81c7dc4e4efa
go: downloading github.com/olekukonko/tablewriter v0.0.5
/tmp/gocache/entgo.io/ent@v0.11.3/cmd/internal/printer/printer.go:16:2: github.com/olekukonko/tablewriter@v0.0.5: Get "https://proxy.golang.org/github.com/olekukonko/tablewriter/@v/v0.0.5.zip": dial tcp: lookup proxy.golang.org: i/o timeout
/tmp/gocache/entgo.io/ent@v0.11.3/entc/load/load.go:27:2: golang.org/x/tools@v0.1.13-0.20220804200503-81c7dc4e4efa: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.13-0.20220804200503-81c7dc4e4efa.zip": dial tcp: lookup proxy.golang.org: i/o timeout
/tmp/gocache/entgo.io/ent@v0.11.3/entc/gen/graph.go:25:2: golang.org/x/tools@v0.1.13-0.20220804200503-81c7dc4e4efa: Get "https://proxy.golang.org/golang.org/x/tools/@v/v0.1.13-0.20220804200503-81c7dc4e4efa.zip": dial tcp: lookup proxy.golang.org: i/o timeout
storage/ent/generate.go:3: running "go": exit status 1

reproducer (with go1.19.2):

Prepare Cash:

git clone https://github.com/dexidp/dex
cd dex
git checkout v2.35.1
env GOMODCACHE=$(realpath ../gocache) go mod download
cd ..

containerfile:

FROM golang:1.19.1-alpine3.16 AS builder

# alpine-sdk would be needed if we could get past the download step
# RUN apk add alpine-sdk

COPY ./dex /src/dex
WORKDIR /src/dex

ARG GOMODCACHE=/tmp/gocache

RUN export GOMODCACHE="$GOMODCACHE" && \
    export GOOS=linux && \
    go generate -mod=mod /src/dex/storage/ent/

Make:

sudo podman network create --internal internal
sudo podman build --network=internal -v $(realpath ./gocache):/tmp/gocache:Z --build-arg GOMODCACHE=/tmp/gocache .

feasible solution?

The ultimate goal is to make the container as secure and reproducible as possible. In fact, it is part of a build system that does roughly as described: pre-downloads dependencies and then builds containers with network isolation.

With that in mind, I’ve found two sub-optimal solutions:

  • Before build, run go generate in addition to the following go mod download
  • Use go mod download all instead (downloads generated dips, not sure why)

Problem with both: the go.sum file for the example project does not include additional dependencies. This means that additional downloads are not verified.

Problem generating Go: The host system does not know what parameters to pass to generate Go. It also may not have the various non-golang dependencies needed to generate it successfully.

Problem downloading all: Downloads even more dependencies, most of which aren’t even needed for Go generates.

Is there any other way I missed downloading it? go generate dependency? The ideal solution would be to include them in go.mod And go.sum of the project. However, it detracts from the normal golang dev workflow. even running go mod tidy removes them again.


Leave a Comment