diff --git a/README.md b/README.md index 9aafaa2..1eb6ead 100644 --- a/README.md +++ b/README.md @@ -1,76 +1,78 @@ -# multikf -Multi-Kind leverages [Vagrant](https://github.com/hashicorp/vagrant) and [Kind](https://github.com/kubernetes-sigs/kind) (Kubernetes In Docker) to create multiple local kubernetes and kubeflow clusters inside the same host machine, see the following png for simple layout -![flow](./images/intro.png) +# multikf +Multi-Kind leverages [Vagrant](https://github.com/hashicorp/vagrant) and [Kind](https://github.com/kubernetes-sigs/kind) (Kubernetes In Docker) to create multiple local kubernetes and kubeflow clusters inside the same host machine, see the following png for simple layout +![flow](./images/intro.png) -#### Why we need this? +#### Why we need this? -As a machine gets more powerful, it is such a waste to have it running just one Kubernetes, especially for the applications which require only a local Kubernetes for practice. One example is our [Kubeflow workshop](https://github.com/footprintai/kubeflow-workshop). -To fully utilize hardware resources, we leverage vagrant to construct a fully isolated environment and install required packages on it (e.g. Kubernetes and Kubeflow and more ...), map ports for kubeApi and ssh, and also export its kubeconfg to host. Therefore, users on the host machine can easily talk to the guest Kube-API via kubectl. +As a machine gets more powerful, it is such a waste to have it running just one Kubernetes, especially for the applications which require only a local Kubernetes for practice. One example is our [Kubeflow workshop](https://github.com/footprintai/kubeflow-workshop). To fully utilize hardware resources, we leverage vagrant to construct a fully isolated environment and install required packages on it (e.g. Kubernetes and Kubeflow and more ...), map ports for kubeApi and ssh, and also export its kubeconfg to host. Therefore, users on the host machine can easily talk to the guest Kube-API via kubectl. -#### When we need this? +#### When we need this? -We expected the user are under: +We expected the user are under: +- Windows environemnt with docker desktop installed +- Linux environment -- Windows environemnt with docker desktop installed -- Linux environment +and this tool provides abstractions for them to operate clusters. -and this tool provides abstractions for them to operate clusters. +#### Why Vagrant is required? -#### Why Vagrant is required? +Idealy, we could just use Kind which running as a container to provide resource isolation. However, Kind was unable to isolate resources from its underlying kubelet(see [issue](https://github.com/kubernetes-sigs/kind/issues/877)) due to kubelet's implementation. Thus, Vagrant is served as a resource isolation and provide clean guest enviornment. -Idealy, we could just use Kind which running as a container to provide resource isolation. However, Kind was unable to isolate resources from its underlying kubelet(see [issue](https://github.com/kubernetes-sigs/kind/issues/877)) due to kubelet's implementation. Thus, Vagrant is served as a resource isolation and provide clean guest enviornment. - -NOTE: Vagrant is not battle-tested, so use it with your cautions. - -#### How to use? +NOTE: Vagrant is not battle-tested, so use it with your cautions. +#### How to use? ``` Usage: multikf add [flags] Flags: - --cpus int number of cpus allocated to the guest machine (default 1) - --export_ports string export ports to host, delimited by comma(example: 8443:443 stands for mapping host port 8443 to container port 443) - --f force to create instance regardless the machine status - -h, --help help for add - --memoryg int number of memory in gigabytes allocated to the guest machine (default 1) - --use_gpus int use gpu resources (default: 0), possible value (0 or 1) - --with_ip string with a specific ip address for kubeapi (default: 0.0.0.0) (default "0.0.0.0") - --with_kubeflow install kubeflow modules (default: true) (default true) - --with_password string with a specific password for default user (default: 12341234) (default "12341234") + --cpus int number of cpus allocated to the guest machine (default 1) + --export_ports string export ports to host, delimited by comma(example: 8443:443 stands for mapping host port 8443 to container port 443) + --f force to create instance regardless the machine status + -h, --help help for add + --memoryg int number of memory in gigabytes allocated to the guest machine (default 1) + --use_gpus int use gpu resources (default: 0), possible value (0 or 1) + --with_ip string with a specific ip address for kubeapi (default: 0.0.0.0) (default "0.0.0.0") + --with_kubeflow install kubeflow modules (default: true) (default true) + --with_password string with a specific password for default user (default: 12341234) (default "12341234") + --with_registry_mirrors string configure registry mirrors, format: source|mirror:username:password,source2|mirror2 ``` -##### Add a vagrant machine named test000 with 1 cpu and 1G memory. - +##### Add a vagrant machine named test000 with 1 cpu and 1G memory. ``` ./multikf add test000 --cpus 1 --memoryg 1 --provisioner=vagrant ``` - - ##### Add a docker machine named test001 with 1 cpu, 1G memory, and all gpus. +##### Add a docker machine named test001 with 1 cpu, 1G memory, and all gpus. ``` ./multikf add test000 --cpus=1 --memoryg=1 --use_gpus=1 --provisioner=docker ``` - - - ##### Add a docker machine named test002 with 1 cpu, 1G memory, and password for helloworld. +##### Add a docker machine named test002 with 1 cpu, 1G memory, and password for helloworld. ``` ./multikf add test000 --cpus=1 --memoryg=16 --with_password=helloworld --provisioner=docker ``` -##### Export a vargant machine's kubeconfig +##### Add a docker machine with registry mirrors to a private registry ``` -./multikf export test000 --kubeconfig_path /tmp/test000.kubeconfig - -run kubectl from host - - kubectl get pods --all-namespaces --kubeconfig=/tmp/test000.kubeconfig +./multikf add test003 --cpus=1 --memoryg=1 --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror" ``` +##### Add a docker machine with multiple registry mirrors and authentication +``` +./multikf add test004 --cpus=1 --memoryg=1 --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror:username:password,k8s.gcr.io|https://reg.footprint-ai.com/k8s-mirror" +``` -##### list machines +##### Export a vargant machine's kubeconfig +``` +./multikf export test000 --kubeconfig_path /tmp/test000.kubeconfig +run kubectl from host + +kubectl get pods --all-namespaces --kubeconfig=/tmp/test000.kubeconfig +``` + +##### list machines ``` ./multikf list @@ -81,23 +83,50 @@ run kubectl from host +---------+------------------+---------+------+---------------+ ``` -##### delete a machine - +##### delete a machine ``` ./multikf delete test000 +``` +#### connect a machine +``` +./multikf connect kubeflow test000 ``` -#### connect a machine +#### Registry Mirrors + +You can configure registry mirrors to pull container images from your private registry instead of public registries like Docker Hub. This is useful for: + +- Improving pull speeds by using a local mirror +- Working in air-gapped environments +- Rate limit mitigation +- Using custom private registries +##### Examples of Registry Mirror Configurations: + +###### Basic mirroring (mirror docker.io to a private registry) +``` +./multikf add my-cluster --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror" +``` + +###### Mirror with authentication +``` +./multikf add my-cluster --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror:username:password" ``` -./multikf connect kubeflow test000 +###### Multiple registry mirrors with different projects +``` +./multikf add my-cluster --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror,k8s.gcr.io|https://reg.footprint-ai.com/k8s-mirror" ``` -#### Roadmap +The format is `source|mirror:username:password` where: +- `source`: The source registry (e.g., docker.io, k8s.gcr.io) +- `mirror`: Your private registry URL (including any project path) +- `username:password`: Optional authentication credentials -Fields listed here is on our roadmap. +#### Roadmap + +Fields listed here is on our roadmap. | Fields | machine(Docker) | machine(Vagrant) | |------|------|------| @@ -105,16 +134,13 @@ Fields listed here is on our roadmap. | Memory Isolation | O | O | | GPU Isolation | O | X | | Expose KubeApi IP | O | O | + +#### Gpu Passthough - -#### Gpu Passthough - -For passing gpu to docker container, one approach is to use `--gpus=all` when you launched docker container like. +For passing gpu to docker container, one approach is to use `--gpus=all` when you launched docker container like. ``` docker run -it --gpus=all ubuntu:21.10 /bin/bash - ``` -where it relies on the host's cuda driver. -However, Kind are NOT supported this approach, see [issue](https://github.com/kubernetes-sigs/kind/pull/1886) -However, we use our [home-crafted kind](https://github.com/footprintai/kind/tree/gpu) for this purpose. + +where it relies on the host's cuda driver. However, Kind are NOT supported this approach, see [issue](https://github.com/kubernetes-sigs/kind/pull/1886) However, we use our [home-crafted kind](https://github.com/footprintai/kind/tree/gpu) for this purpose. diff --git a/cmd/multikf/cmd_add.go b/cmd/multikf/cmd_add.go index 5f8efb2..16bcdb5 100644 --- a/cmd/multikf/cmd_add.go +++ b/cmd/multikf/cmd_add.go @@ -33,6 +33,7 @@ func NewAddCommand(logger log.Logger, ioStreams genericclioptions.IOStreams) *co useLocalPath string // with localpath withK8sVersion string withK8sSHA256 string + withRegistryMirrors string // with registry mirrors ) ensureNoGPUForVagrant := func(vag machine.MachineCURDFactory, useGPUs int) error { @@ -55,17 +56,18 @@ func NewAddCommand(logger log.Logger, ioStreams genericclioptions.IOStreams) *co } m, err := vag.NewMachine(machineName, machineConfig{ - logger: logger, - Cpus: cpus, - MemoryInG: memoryInG, - UseGPUs: useGPUs, - KubeAPIIP: withIP, - ExportPorts: exportPorts, - ForceOverwrite: forceOverwrite, - IsAuditEnabled: withAudit, - Workers: withWorkers, - NodeLabels: withLabels, - LocalPath: useLocalPath, + logger: logger, + Cpus: cpus, + MemoryInG: memoryInG, + UseGPUs: useGPUs, + KubeAPIIP: withIP, + ExportPorts: exportPorts, + ForceOverwrite: forceOverwrite, + IsAuditEnabled: withAudit, + Workers: withWorkers, + NodeLabels: withLabels, + LocalPath: useLocalPath, + RegistryMirrors: withRegistryMirrors, NodeVersion: k8s.NewKindK8sVersion( withK8sVersion, withK8sSHA256, @@ -111,6 +113,7 @@ func NewAddCommand(logger log.Logger, ioStreams genericclioptions.IOStreams) *co cmd.Flags().StringVar(&useLocalPath, "use_localpath", "", "mount local path to kind cluster") cmd.Flags().StringVar(&withK8sVersion, "with_k8s_version", k8s.DefaultVersion().Version(), fmt.Sprintf("support verisions:%s", strings.Join(k8s.ListVersionString(), ","))) cmd.Flags().StringVar(&withK8sSHA256, "with_k8s_sha256", k8s.DefaultVersion().Sha256(), fmt.Sprintf("k8s version and its sha256 mapping list:%s", strings.Join(k8s.ListVersionSha256String(), ","))) + cmd.Flags().StringVar(&withRegistryMirrors, "with_registry_mirrors", "", "configure registry mirrors, format: source|mirror:username:password,source2|mirror2 (example: docker.io|https://reg.footprint-ai.com/kubeflow-mirror:username:password)") return cmd } diff --git a/cmd/multikf/helper.go b/cmd/multikf/helper.go index 94a4a90..ba63a06 100644 --- a/cmd/multikf/helper.go +++ b/cmd/multikf/helper.go @@ -9,6 +9,7 @@ import ( "github.com/footprintai/multikf/pkg/k8s" "github.com/footprintai/multikf/pkg/machine" "github.com/footprintai/multikf/pkg/machine/plugins" + "github.com/footprintai/multikf/pkg/mirror" "sigs.k8s.io/kind/pkg/log" ) @@ -78,12 +79,12 @@ type machineConfig struct { NodeLabels string `json:"node_labels"` LocalPath string `json:"local_path"` NodeVersion k8s.KindK8sVersion `json:"node_version"` + RegistryMirrors string `json:"registry_mirrors"` // New field for registry mirrors } func (m machineConfig) Info() string { bb, _ := json.Marshal(m) return string(bb) - } func (m machineConfig) GetNodeVersion() k8s.KindK8sVersion { @@ -172,6 +173,64 @@ func (m machineConfig) GetLocalPath() string { return m.LocalPath } +// GetRegistry implements the mirror.Getter interface +// Format: source|mirror1,mirror2:username:password,source2|mirror3,mirror4 +// Example: docker.io|https://reg.footprint-ai.com/kubeflow-mirror:username:password,k8s.gcr.io|https://reg.footprint-ai.com/k8s-mirror +func (m machineConfig) GetRegistry() []mirror.Registry { + if len(m.RegistryMirrors) == 0 { + m.logger.V(1).Infof("getregistry: no registry mirrors\n") + return nil + } + + registryEntries := strings.Split(m.RegistryMirrors, ",") + var registries []mirror.Registry + + for _, entry := range registryEntries { + parts := strings.Split(entry, "|") + if len(parts) != 2 { + m.logger.Errorf("getregistry: parse failed, expect: source|mirrors but got:%s\n", entry) + continue + } + + source := parts[0] + mirrorParts := strings.Split(parts[1], ":") + + // Check if authentication is provided + var auth *mirror.Auth + var mirrors []string + + if len(mirrorParts) >= 3 { + // Format with auth: mirror:username:password + mirrors = []string{mirrorParts[0]} + auth = &mirror.Auth{ + Username: mirrorParts[1], + Password: mirrorParts[2], + } + } else if len(mirrorParts) == 1 { + // Format without auth: just mirror + mirrors = []string{mirrorParts[0]} + auth = nil + } else { + m.logger.Errorf("getregistry: parse failed, invalid mirror format: %s\n", parts[1]) + continue + } + + registries = append(registries, mirror.Registry{ + Source: source, + Mirrors: mirrors, + Auth: auth, + }) + } + + m.logger.V(1).Infof("getregistry: registry mirrors:%+v\n", registries) + return registries +} + +func AuditFileAbsolutePath() string { + // Return the path directly if already set somewhere else in your code + return "/path/to/audit/policy.yaml" +} + type kubeflowPlugin struct { withKubeflowDefaultPassword string kubeflowVersion plugins.TypePluginVersion @@ -183,7 +242,6 @@ func (k kubeflowPlugin) PluginType() plugins.TypePlugin { func (k kubeflowPlugin) PluginVersion() plugins.TypePluginVersion { return k.kubeflowVersion - } func (k kubeflowPlugin) GetDefaultPassword() string { diff --git a/pkg/machine/docker/hostmachine.go b/pkg/machine/docker/hostmachine.go index 1918162..9c6a920 100644 --- a/pkg/machine/docker/hostmachine.go +++ b/pkg/machine/docker/hostmachine.go @@ -136,6 +136,7 @@ func (h *HostMachine) prepareFiles() error { h.options.GetNodeLabels(), h.options.GetLocalPath(), h.options.GetNodeVersion(), + h.options.GetRegistry(), ) vfolder := NewHostFolder(h.hostMachineDir) diff --git a/pkg/machine/docker/template/config.go b/pkg/machine/docker/template/config.go index 41b2efa..4116d08 100644 --- a/pkg/machine/docker/template/config.go +++ b/pkg/machine/docker/template/config.go @@ -3,6 +3,7 @@ package template import ( "github.com/footprintai/multikf/pkg/k8s" "github.com/footprintai/multikf/pkg/machine" + "github.com/footprintai/multikf/pkg/mirror" pkgtemplateconfig "github.com/footprintai/multikf/pkg/template/config" ) @@ -10,7 +11,23 @@ type DockerHostmachineTemplateConfig struct { *pkgtemplateconfig.DefaultTemplateConfig } -func NewDockerHostmachineTemplateConfig(name string, cpus int, memory int, sshport int, kubeApiPort int, kubeApiIP string, gpus int, exportPorts []machine.ExportPortPair, auditEnabled bool, auditFileAbsolutePath string, workerCount int, nodeLabels []machine.NodeLabel, localPath string, nodeVersion k8s.KindK8sVersion) *DockerHostmachineTemplateConfig { +func NewDockerHostmachineTemplateConfig( + name string, + cpus int, + memory int, + sshport int, + kubeApiPort int, + kubeApiIP string, + gpus int, + exportPorts []machine.ExportPortPair, + auditEnabled bool, + auditFileAbsolutePath string, + workerCount int, + nodeLabels []machine.NodeLabel, + localPath string, + nodeVersion k8s.KindK8sVersion, + registryMirrors []mirror.Registry, +) *DockerHostmachineTemplateConfig { return &DockerHostmachineTemplateConfig{ DefaultTemplateConfig: pkgtemplateconfig.NewDefaultTemplateConfig( name, @@ -27,6 +44,7 @@ func NewDockerHostmachineTemplateConfig(name string, cpus int, memory int, sshpo nodeLabels, localPath, nodeVersion, + registryMirrors, ), } } diff --git a/pkg/machine/machine.go b/pkg/machine/machine.go index a2dbab8..530a6ae 100644 --- a/pkg/machine/machine.go +++ b/pkg/machine/machine.go @@ -3,6 +3,7 @@ package machine import ( "github.com/footprintai/multikf/pkg/k8s" "github.com/footprintai/multikf/pkg/machine/cmd/kubectl" + "github.com/footprintai/multikf/pkg/mirror" ) type MachineCURDFactory interface { @@ -23,7 +24,7 @@ type MachineConfiger interface { GetNodeLabels() []NodeLabel GetLocalPath() string GetNodeVersion() k8s.KindK8sVersion - + mirror.Getter // Embed the mirror.Getter interface // Info displays all configurations Info() string } @@ -63,9 +64,10 @@ type MachineCURD interface { } type MachineInfo struct { - CpuInfo *CpuInfo - MemInfo *MemInfo - GpuInfo *GpuInfo - KubeApi string - Status string + CpuInfo *CpuInfo + MemInfo *MemInfo + GpuInfo *GpuInfo + KubeApi string + Status string + RegistryMirrors []mirror.Registry // Using the Registry type from mirror package } diff --git a/pkg/machine/vagrant/template/config.go b/pkg/machine/vagrant/template/config.go index 548e28c..789d1c0 100644 --- a/pkg/machine/vagrant/template/config.go +++ b/pkg/machine/vagrant/template/config.go @@ -3,6 +3,7 @@ package template import ( "github.com/footprintai/multikf/pkg/k8s" "github.com/footprintai/multikf/pkg/machine" + "github.com/footprintai/multikf/pkg/mirror" pkgtemplateconfig "github.com/footprintai/multikf/pkg/template/config" ) @@ -10,7 +11,23 @@ type VagrantTemplateConfig struct { *pkgtemplateconfig.DefaultTemplateConfig } -func NewVagrantTemplateConfig(name string, cpus int, memory int, sshport int, kubeApiPort int, kubeApiIP string, gpus int, exportPorts []machine.ExportPortPair, auditEnabled bool, auditFileAbsolutePath string, workerCount int, nodeLabels []machine.NodeLabel, localPath string, nodeVersion k8s.KindK8sVersion) *VagrantTemplateConfig { +func NewVagrantTemplateConfig( + name string, + cpus int, + memory int, + sshport int, + kubeApiPort int, + kubeApiIP string, + gpus int, + exportPorts []machine.ExportPortPair, + auditEnabled bool, + auditFileAbsolutePath string, + workerCount int, + nodeLabels []machine.NodeLabel, + localPath string, + nodeVersion k8s.KindK8sVersion, + registryMirrors []mirror.Registry, +) *VagrantTemplateConfig { return &VagrantTemplateConfig{ DefaultTemplateConfig: pkgtemplateconfig.NewDefaultTemplateConfig( name, @@ -27,6 +44,7 @@ func NewVagrantTemplateConfig(name string, cpus int, memory int, sshport int, ku nodeLabels, localPath, nodeVersion, + registryMirrors, ), } } diff --git a/pkg/machine/vagrant/vagrant.go b/pkg/machine/vagrant/vagrant.go index 2b3e342..576ccd5 100644 --- a/pkg/machine/vagrant/vagrant.go +++ b/pkg/machine/vagrant/vagrant.go @@ -222,6 +222,7 @@ func (v *VagrantMachine) prepareFiles() error { v.options.GetNodeLabels(), v.options.GetLocalPath(), v.options.GetNodeVersion(), + v.options.GetRegistry(), ) vfolder := NewVagrantFolder(v.vagrantMachineDir) diff --git a/pkg/mirror/mirror.go b/pkg/mirror/mirror.go new file mode 100644 index 0000000..38af68d --- /dev/null +++ b/pkg/mirror/mirror.go @@ -0,0 +1,19 @@ +package mirror + +// Auth represents authentication credentials for a registry +type Auth struct { + Username string + Password string +} + +// Registry represents a container registry mirror configuration +type Registry struct { + Source string // The registry to be replaced (e.g., "docker.io") + Mirrors []string // The mirror endpoints (e.g., ["reg.footprint-ai.com"]) + Auth *Auth // Optional authentication information +} + +// Getter defines the interface to get registry mirror configuration +type Getter interface { + GetRegistry() []Registry +} diff --git a/pkg/template/config/config.go b/pkg/template/config/config.go index a62123e..a369415 100644 --- a/pkg/template/config/config.go +++ b/pkg/template/config/config.go @@ -5,6 +5,7 @@ import ( "github.com/footprintai/multikf/pkg/k8s" "github.com/footprintai/multikf/pkg/machine" + "github.com/footprintai/multikf/pkg/mirror" "github.com/footprintai/multikf/pkg/template" ) @@ -27,9 +28,33 @@ type DefaultTemplateConfig struct { nodeLabels []machine.NodeLabel localPath string nodeVersion k8s.KindK8sVersion -} + registryMirrors []mirror.Registry // Using the Registry type from mirror package +} + +// NewDefaultTemplateConfig creates a default template config +// registryMirrors can be nil or an empty slice if no registry mirrors are needed +func NewDefaultTemplateConfig( + name string, + cpus int, + memory int, + sshport int, + kubeApiPort int, + kubeApiIP string, + gpus int, + exportPorts []machine.ExportPortPair, + auditEnabled bool, + auditFileAbsolutePath string, + workerCount int, + nodeLabels []machine.NodeLabel, + localPath string, + nodeVersion k8s.KindK8sVersion, + registryMirrors []mirror.Registry, +) *DefaultTemplateConfig { + // If registryMirrors is nil, initialize as empty slice + if registryMirrors == nil { + registryMirrors = []mirror.Registry{} + } -func NewDefaultTemplateConfig(name string, cpus int, memory int, sshport int, kubeApiPort int, kubeApiIP string, gpus int, exportPorts []machine.ExportPortPair, auditEnabled bool, auditFileAbsolutePath string, workerCount int, nodeLabels []machine.NodeLabel, localPath string, nodeVersion k8s.KindK8sVersion) *DefaultTemplateConfig { return &DefaultTemplateConfig{ name: name, cpus: cpus, @@ -45,9 +70,32 @@ func NewDefaultTemplateConfig(name string, cpus int, memory int, sshport int, ku nodeLabels: nodeLabels, localPath: localPath, nodeVersion: nodeVersion, + registryMirrors: registryMirrors, } } +// AddRegistryMirror adds a registry mirror to the config +func (t *DefaultTemplateConfig) AddRegistryMirror(registry mirror.Registry) { + t.registryMirrors = append(t.registryMirrors, registry) +} + +// AddAuthenticatedRegistryMirror adds a registry mirror with authentication to the config +func (t *DefaultTemplateConfig) AddAuthenticatedRegistryMirror(source string, mirrorURL string, username string, password string) { + t.registryMirrors = append(t.registryMirrors, mirror.Registry{ + Source: source, + Mirrors: []string{mirrorURL}, + Auth: &mirror.Auth{ + Username: username, + Password: password, + }, + }) +} + +// GetRegistry implements the mirror.Getter interface +func (t *DefaultTemplateConfig) GetRegistry() []mirror.Registry { + return t.registryMirrors +} + func (t *DefaultTemplateConfig) GetName() string { return t.name } diff --git a/pkg/template/kind_template.go b/pkg/template/kind_template.go index a7c99dd..f662a88 100644 --- a/pkg/template/kind_template.go +++ b/pkg/template/kind_template.go @@ -6,6 +6,7 @@ import ( "io" "github.com/footprintai/multikf/pkg/machine" + "github.com/footprintai/multikf/pkg/mirror" ) func NewKindTemplate() *KindFileTemplate { @@ -40,6 +41,7 @@ type KindConfiger interface { WorkersGetter NodeLabelsGetter LocalPathGetter + mirror.Getter // Use the Getter interface from mirror package } func (k *KindFileTemplate) Populate(v interface{}) error { @@ -64,6 +66,9 @@ func (k *KindFileTemplate) Populate(v interface{}) error { k.NodeLabels[idx] = fmt.Sprintf("%s=%s", nodeLabels[idx].Key, nodeLabels[idx].Value) } + // Set registry mirrors + k.RegistryMirrors = c.GetRegistry() + return nil } @@ -80,6 +85,7 @@ type KindFileTemplate struct { Workers []Worker NodeLabels []string NodeVersion string + RegistryMirrors []mirror.Registry // Using the Registry type from the mirror package } var ( @@ -125,6 +131,22 @@ nodes: {{- range $i, $p := .NodeLabels}} node-labels: "{{$p}}" {{- end}} + {{- if .RegistryMirrors}} + containerdConfigPatches: + - |- + [plugins."io.containerd.grpc.v1.cri".registry] + config_path = "" + {{- range $mirror := .RegistryMirrors}} + [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{$mirror.Source}}"] + endpoint = [{{- range $i, $endpoint := $mirror.Mirrors}}{{if $i}}, {{end}}"{{$endpoint}}"{{- end}}] + {{- end}} + {{- range $mirror := .RegistryMirrors}} + {{- if $mirror.Auth}} + [plugins."io.containerd.grpc.v1.cri".registry.configs."{{$mirror.Mirrors | first}}"] + auth = { username = "{{$mirror.Auth.Username}}", password = "{{$mirror.Auth.Password}}" } + {{- end}} + {{- end}} + {{- end}} image: {{.NodeVersion}} gpus: {{.UseGPU}} {{if .ExportPorts}}extraPortMappings:{{end}} @@ -149,6 +171,22 @@ nodes: - role: worker image: {{ .NodeVersion}} gpus: {{ .UseGPU}} + {{- if $.RegistryMirrors}} + containerdConfigPatches: + - |- + [plugins."io.containerd.grpc.v1.cri".registry] + config_path = "" + {{- range $mirror := $.RegistryMirrors}} + [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{$mirror.Source}}"] + endpoint = [{{- range $i, $endpoint := $mirror.Mirrors}}{{if $i}}, {{end}}"{{$endpoint}}"{{- end}}] + {{- end}} + {{- range $mirror := $.RegistryMirrors}} + {{- if $mirror.Auth}} + [plugins."io.containerd.grpc.v1.cri".registry.configs."{{$mirror.Mirrors | first}}"] + auth = { username = "{{$mirror.Auth.Username}}", password = "{{$mirror.Auth.Password}}" } + {{- end}} + {{- end}} + {{- end}} {{- if ne .LocalPath ""}} extraMounts: - hostPath: {{.LocalPath}} diff --git a/pkg/template/template.go b/pkg/template/template.go index 0e07404..65eb44d 100644 --- a/pkg/template/template.go +++ b/pkg/template/template.go @@ -6,6 +6,7 @@ import ( "github.com/footprintai/multikf/pkg/k8s" "github.com/footprintai/multikf/pkg/machine" + "github.com/footprintai/multikf/pkg/mirror" ) type TemplateExecutor interface { @@ -60,6 +61,11 @@ type WorkersGetter interface { GetWorkers() []Worker } +// RegistryGetter defines the interface to get registry mirror configuration +type RegistryGetter interface { + GetRegistryMirrors() []mirror.Registry +} + type Worker struct { Id string UseGPU bool diff --git a/scripts/mirror/READMD.md b/scripts/mirror/READMD.md new file mode 100644 index 0000000..a4eb65c --- /dev/null +++ b/scripts/mirror/READMD.md @@ -0,0 +1,160 @@ +# Container Image Mirror Tool + +A Python utility for mirroring container images from public registries to private registries. This tool makes it easy to create a local mirror of required images, which is useful for air-gapped environments, rate limit mitigation, or speeding up deployments with a local cache. + +## Features + +- Mirror individual container images or process images in batch +- Support for authenticated registry access +- Preserves image path structure in the target registry +- Handles various image naming formats (with or without explicit registry, repo paths, tags, or digests) +- Option to execute commands directly or just print them for manual execution + +## Installation + +No special installation is required beyond Python 3. Simply download the script and make it executable: + +```bash +chmod +x mirror_image.py +``` + +## Usage + +### Basic Usage + +Mirror a single image: + +```bash +./mirror_image.py --image docker.io/kubeflow/training-operator:v1.5.0 --mirror reg.footprint-ai.com/kubeflow-mirror +``` + +This will output the commands needed to pull, tag, and push the image: + +```bash +docker pull docker.io/kubeflow/training-operator:v1.5.0 +docker tag docker.io/kubeflow/training-operator:v1.5.0 reg.footprint-ai.com/kubeflow-mirror/kubeflow/training-operator:v1.5.0 +docker push reg.footprint-ai.com/kubeflow-mirror/kubeflow/training-operator:v1.5.0 +``` + +### Mirror with Authentication + +If your private registry requires authentication: + +```bash +./mirror_image.py --image docker.io/kubeflow/training-operator:v1.5.0 --mirror reg.footprint-ai.com/kubeflow-mirror --username myuser --password mypass +``` + +This will add a login command before the other operations: + +```bash +echo mypass | docker login reg.footprint-ai.com/kubeflow-mirror --username myuser --password-stdin +docker pull docker.io/kubeflow/training-operator:v1.5.0 +docker tag docker.io/kubeflow/training-operator:v1.5.0 reg.footprint-ai.com/kubeflow-mirror/kubeflow/training-operator:v1.5.0 +docker push reg.footprint-ai.com/kubeflow-mirror/kubeflow/training-operator:v1.5.0 +``` + +### Batch Processing + +To mirror multiple images, create a text file with one image per line: + +``` +# images.txt +docker.io/kubeflow/training-operator:v1.5.0 +docker.io/kubeflow/katib-controller:v0.15.0 +k8s.gcr.io/kube-scheduler:v1.21.0 +quay.io/coreos/kube-state-metrics:v1.9.7 +nginx:latest +``` + +Then process the batch file: + +```bash +./mirror_image.py --batch-file images.txt --mirror reg.footprint-ai.com/kubeflow-mirror +``` + +### Execute Commands + +To directly execute the commands instead of just printing them: + +```bash +./mirror_image.py --image docker.io/kubeflow/training-operator:v1.5.0 --mirror reg.footprint-ai.com/kubeflow-mirror --execute +``` + +Output: + +``` +Executing: docker pull docker.io/kubeflow/training-operator:v1.5.0 +Executing: docker tag docker.io/kubeflow/training-operator:v1.5.0 reg.footprint-ai.com/kubeflow-mirror/kubeflow/training-operator:v1.5.0 +Executing: docker push reg.footprint-ai.com/kubeflow-mirror/kubeflow/training-operator:v1.5.0 +``` + +## Command Line Arguments + +| Argument | Description | +|----------|-------------| +| `--image` | Source image path (e.g., docker.io/kubeflow/training-operator:v1.5.0) | +| `--mirror` | Mirror registry (e.g., reg.footprint-ai.com/kubeflow-mirror) | +| `--username` | Registry username for authentication | +| `--password` | Registry password for authentication | +| `--batch-file` | File containing list of images to mirror (one per line) | +| `--execute` | Execute the commands instead of printing them | + +## Image Name Handling + +The tool handles various image formats: + +- Images with explicit registry: `docker.io/kubeflow/training-operator:v1.5.0` +- Images with implicit registry: `kubeflow/training-operator:v1.5.0` (assumes docker.io) +- Official images: `nginx:latest` (assumes docker.io/library) +- Images with digest: `docker.io/kubeflow/training-operator@sha256:123abc...` +- Images without tag: `kubeflow/training-operator` (assumes :latest tag) +- Images with multi-level paths: `docker.io/kubeflow/common/katib-controller:v0.15.0` + +## Integration with multikf + +This tool complements the registry mirror feature in multikf. You can: + +1. Use this script to populate your private registry with required images +2. Configure multikf to use your mirror registry: + +```bash +multikf add my-cluster --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror" +``` + +## Use Cases + +### Preparing for Air-gapped Deployments + +```bash +# Mirror all required images +./mirror_image.py --batch-file required-images.txt --mirror registry.internal --execute + +# Configure multikf to use the mirror +multikf add airgap-cluster --with_registry_mirrors="docker.io|https://registry.internal,k8s.gcr.io|https://registry.internal/k8s-mirror" +``` + +### Working Around Rate Limits + +```bash +# Mirror images that might hit rate limits +./mirror_image.py --batch-file frequently-used-images.txt --mirror reg.footprint-ai.com/cache --execute + +# Configure multikf to use the mirror +multikf add dev-cluster --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/cache" +``` + +### Creating Project-specific Image Bundles + +```bash +# Mirror project-specific images to a dedicated project path +./mirror_image.py --batch-file kubeflow-images.txt --mirror reg.footprint-ai.com/kubeflow-mirror --execute + +# Configure multikf to use the project-specific mirror +multikf add kf-cluster --with_registry_mirrors="docker.io|https://reg.footprint-ai.com/kubeflow-mirror" +``` + +## License + +[Apache License 2.0](LICENSE) + +This project is licensed under the Apache License, Version 2.0. See the LICENSE file for the full license text. diff --git a/scripts/mirror/image_mirror.py b/scripts/mirror/image_mirror.py new file mode 100644 index 0000000..ee33171 --- /dev/null +++ b/scripts/mirror/image_mirror.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +""" +Image Mirror Tool - Creates commands to mirror container images to private registries. + +Usage: + python mirror_image.py --image docker.io/kubeflow/training-operator:v1.5.0 --mirror reg.footprint-ai.com/kubeflow-mirror + + Optional arguments: + --username USERNAME - Registry username for authentication + --password PASSWORD - Registry password for authentication + --batch-file FILE - File containing list of images to mirror (one per line) + --execute - Execute the generated commands instead of just printing them +""" + +import argparse +import subprocess +import sys +import os +from typing import List, Optional, Tuple + +def parse_image_name(image_path: str) -> Tuple[str, str, str, str]: + """ + Parse the image path into its components. + + Args: + image_path: Full image path (e.g., docker.io/kubeflow/training-operator:v1.5.0) + + Returns: + Tuple of (registry, repository, image, tag) + """ + # Handle registry with port if present + if '@' in image_path: + # Handle digest format + path_parts, digest = image_path.split('@') + tag = f"@{digest}" + elif ':' in image_path.split('/')[-1]: + # Handle normal tag format + path_parts, tag = image_path.rsplit(':', 1) + tag = f":{tag}" + else: + # Default to latest if no tag is specified + path_parts = image_path + tag = ":latest" + + # Split the remaining path + parts = path_parts.split('/') + + # Handle different path formats + if len(parts) == 1: + # Image only, assume docker.io/library + registry = "docker.io" + repo = "library" + image = parts[0] + elif len(parts) == 2: + # Could be either registry/image or namespace/image + if '.' in parts[0] or parts[0] == "localhost": + # It's registry/image + registry = parts[0] + repo = "" + image = parts[1] + else: + # It's namespace/image on default registry + registry = "docker.io" + repo = parts[0] + image = parts[1] + else: + # registry/namespace/image or registry/namespace/subnamespace/image + registry = parts[0] + image = parts[-1] + repo = '/'.join(parts[1:-1]) + + return registry, repo, image, tag + +def generate_mirror_commands(image_path: str, mirror_registry: str, username: Optional[str] = None, + password: Optional[str] = None) -> List[str]: + """ + Generate commands to pull, tag and push an image to a mirror registry. + + Args: + image_path: Full image path + mirror_registry: Target mirror registry + username: Optional registry username + password: Optional registry password + + Returns: + List of shell commands to execute + """ + commands = [] + + # Parse the image name + source_registry, repo, image, tag = parse_image_name(image_path) + + # Build the full source path + if repo: + source_path = f"{source_registry}/{repo}/{image}{tag}" + else: + source_path = f"{source_registry}/{image}{tag}" + + # Build the destination path + if repo: + dest_path = f"{mirror_registry}/{repo}/{image}{tag}" + else: + dest_path = f"{mirror_registry}/{image}{tag}" + + # Add login command if credentials are provided + if username and password: + commands.append(f"echo {password} | docker login {mirror_registry} --username {username} --password-stdin") + + # Add pull, tag and push commands + commands.append(f"docker pull {source_path}") + commands.append(f"docker tag {source_path} {dest_path}") + commands.append(f"docker push {dest_path}") + + return commands + +def process_batch_file(batch_file: str, mirror_registry: str, username: Optional[str] = None, + password: Optional[str] = None, execute: bool = False) -> None: + """ + Process a batch file containing one image per line. + + Args: + batch_file: Path to file with image list + mirror_registry: Target mirror registry + username: Optional registry username + password: Optional registry password + execute: Whether to execute commands + """ + if not os.path.exists(batch_file): + print(f"Error: Batch file {batch_file} not found.") + sys.exit(1) + + with open(batch_file, 'r') as f: + images = [line.strip() for line in f if line.strip() and not line.strip().startswith('#')] + + print(f"Processing {len(images)} images from {batch_file}...") + + # Add login command once if credentials are provided + if username and password and execute: + login_cmd = f"echo {password} | docker login {mirror_registry} --username {username} --password-stdin" + print(f"Executing: {login_cmd}") + subprocess.run(login_cmd, shell=True, check=True) + + for image in images: + print(f"\nMirroring: {image}") + commands = generate_mirror_commands(image, mirror_registry, None, None) # Skip login command for batch + + if execute: + for cmd in commands: + print(f"Executing: {cmd}") + try: + subprocess.run(cmd, shell=True, check=True) + except subprocess.CalledProcessError as e: + print(f"Error executing command: {e}") + else: + for cmd in commands: + print(cmd) + +def main() -> None: + parser = argparse.ArgumentParser(description="Generate commands to mirror container images") + parser.add_argument("--image", help="Source image path (e.g., docker.io/kubeflow/training-operator:v1.5.0)") + parser.add_argument("--mirror", required=True, help="Mirror registry (e.g., reg.footprint-ai.com/kubeflow-mirror)") + parser.add_argument("--username", help="Registry username for authentication") + parser.add_argument("--password", help="Registry password for authentication") + parser.add_argument("--batch-file", help="File containing list of images to mirror (one per line)") + parser.add_argument("--execute", action="store_true", help="Execute the commands instead of printing them") + + args = parser.parse_args() + + if args.batch_file: + process_batch_file(args.batch_file, args.mirror, args.username, args.password, args.execute) + elif args.image: + commands = generate_mirror_commands(args.image, args.mirror, args.username, args.password) + + if args.execute: + for cmd in commands: + print(f"Executing: {cmd}") + try: + subprocess.run(cmd, shell=True, check=True) + except subprocess.CalledProcessError as e: + print(f"Error executing command: {e}") + else: + print("\n".join(commands)) + else: + parser.print_help() + print("\nError: Either --image or --batch-file must be specified.") + sys.exit(1) + +if __name__ == "__main__": + main()