How Go Finds Std Libs¶

Usually, the go standard library is located under $GOROOT and it's go compiler's scope to find the std libs, so most of the users needn't care about how the standard library is found by compiler.

When we want to combine 2 services into a single process, we need to solve the collision of global states between 2 services in the standard library. By duplicating it with another name(and all underlying dependent packages) under GOROOT, we can segregate the global states well even though they refer to the same package before combining. However, this requires to investigate how go compiler finds the std lib to ensure the approach won't break.

What's the meaning of duplicating packages?¶

For example, previously service1 and service 2 both assign the global client under http package.

Separated Codebase

package service1

import (
    "net/http"
)

func Service1Handler() {
    http.DefaultClient = http.Client{ /* custom logic 1 */}
}

package service2

import (
    "net/http"
)
func Service2Handler() {
    http.DefaultClient = http.Client{ /* custom logic 2 */}
}

If we combine 2 services by building them together, the assignments to http.DefaultClient will intervene with each other. The duplication means I will fork the net/http and create another std package like the code below. By this way, the DefaultClient locates in different packages so the assignments won't affect each other.

Duplicate Std Lib

package service1

import (
    "net/http"
)

func Service1Handler() {
    http.DefaultClient = http.Client{ /* custom logic 1 */}
}

package service2
import (
    fhttp "forked/net/http"
)
func Service2Handler() {
    fhttp.DefaultClient = fhttp.Client{ /* custom logic 2 */}
}

How go compiler finds std lib¶

Function isStandardImportPath helps to check whether the path refers to a standard library.

Go compiler will reject any path with dot as a standard library. Then it tries to find the path under GOROOT/src as mod root. If it's a package under cmd such as cmd/go/internal/base, it tries to find it under GOROOT/src/cmd as mod root.

The source code contains a GetPackage but it doesn't apply to the standard libraries. So the logic of standard library always relies on root.IsStandardPackage which checks the path under $GOROOT/src.

var errNotFromModuleCache = 
    fmt.Errorf("%w: not from module cache", ErrNotIndexed)

// the original code is one line, I split it in 2 lines to render them better.
if str.HasFilePathPrefix(modroot, cfg.GOROOTsrc) 
    || !str.HasFilePathPrefix(modroot, cfg.GOMODCACHE) {
    return nil, errNotFromModuleCache
}

Go Searching Standard Library Code

// src/cmd/go/internal/modload/build.go#L34
func isStandardImportPath(path string) bool {
    return findStandardImportPath(path) != ""
}

func findStandardImportPath(path string) string {
    if path == "" {
        panic("findStandardImportPath called with empty path")
    }
    if search.IsStandardImportPath(path) {
        if modindex.IsStandardPackage(cfg.GOROOT, cfg.BuildContext.Compiler, path) {
            return filepath.Join(cfg.GOROOT, "src", path)
        }
    }
    return ""
}

// src/cmd/go/internal/search/search.go#L452
func IsStandardImportPath(path string) bool {
    i := strings.Index(path, "/")
    if i < 0 {
        i = len(path)
    }
    elem := path[:i]
    return !strings.Contains(elem, ".")
}


// cmd/go/internal/modindex/read.go
package modindex
func IsStandardPackage(goroot_, compiler, path string) bool {
    if !enabled || compiler != "gc" {
        return goroot.IsStandardPackage(goroot_, compiler, path)
    }

    reldir := filepath.FromSlash(path) // relative dir path in module index for package
    modroot := filepath.Join(goroot_, "src")
    if str.HasFilePathPrefix(reldir, "cmd") {
        reldir = str.TrimFilePathPrefix(reldir, "cmd")
        modroot = filepath.Join(modroot, "cmd")
    }
    if pkg, err := GetPackage(modroot, filepath.Join(modroot, reldir)); err == nil {
        hasGo, err := pkg.IsDirWithGoFiles()
        return err == nil && hasGo
    } else if errors.Is(err, ErrNotIndexed) {
        // Fall back because package isn't indexable. (Probably because
        // a file was modified recently)
        return goroot.IsStandardPackage(goroot_, compiler, path)
    }
    return false
}

// src/internal/goroot/gc.go
package goroot
func IsStandardPackage(goroot, compiler, path string) bool {
    switch compiler {
    case "gc":
        dir := filepath.Join(goroot, "src", path)
        dirents, err := os.ReadDir(dir)
        if err != nil {
            return false
        }
        for _, dirent := range dirents {
            if strings.HasSuffix(dirent.Name(), ".go") {
                return true
            }
        }
        return false
    case "gccgo":
        return gccgoSearch.isStandard(path)
    default:
        panic("unknown compiler " + compiler)
    }
}

Chore: View some variables¶

By inserting a printf, we can view the paths.

--- a/src/cmd/go/internal/modindex/read.go
+++ b/src/cmd/go/internal/modindex/read.go
@@ -670,6 +670,8@@ func IsStandardPackage(goroot_, compiler, path string) bool {
                reldir = str.TrimFilePathPrefix(reldir, "cmd")
                modroot = filepath.Join(modroot, "cmd")
        }
+       fmt.Printf("GetPakcage: \n\tmodroot: %s\n\tpath: %s\n\treldir: %s\n",
+               modroot, path, reldir)

The repo is cloned at /Users/yuchen.xie/workspace/golang/go.

GetPakcage: 
        modroot: /Users/yuchen.xie/workspace/golang/go/src/cmd
        path: cmd/link/internal/loadpe
        reldir: link/internal/loadpe  
GetPakcage: 
        modroot: /Users/yuchen.xie/workspace/golang/go/src
        path: debug/buildinfo
        reldir: debug/buildinfo

Conclusion¶

It's safe to duplicate the std lib under GOROOT/src because go just checks whether the path exists or not and nothing more.