[TOC]

1. 目录结构

以kubernetes 1.22版本代码为例，kubelet的核心源码在cmd/kubelet、pkg/kubelet下，其中cmd/kubelet为main函数入口，主要负责参数加载、运行框架构建等前期准备工作逻辑，pkg/kubelet为kubelet主要功能细节运行逻辑代码，如下图分别是代码目录结构：

cmd/kubelet代码目录.png

pkg/kubelet代码目录.png

2. 程序入口

kubelet的main函数文件路径：kubernetes/cmd/kubelet/kubelet.go，cmd下是各类组件(如：kubectl、kubelet、kube-apiserver等)的程序入口，都是通过cobra工具生成的脚手架，其中kubernetes/cmd/kubelet/下app目录则是通过cobra生成的，后续我们将简单聊一聊cobra工具

image.png

3. cobra工具

cobra是基于go的开源代码脚手架工具，网上资料很多，k8s的组件代码基本都是通过该工具生成，所以我们这里就拣一些基本使用流程说说，不深入细节了，主要从以下几个点来聊：

• cobra使用
• 生成的代码框架
• 程序入口及调用逻辑
• Run函数解析

3.1 cobra使用

cobra工具需要下载安装，安装后通过命令生成代码框架，通过cobra -h我们能看到，cobra有init和add两个子命令，init即从0生成自己的程序主代码结构，add则是添加程序子命令或分支：

$ cobra -h
Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Usage:
  cobra [command]

Available Commands:
  add         Add a command to a Cobra Application
  help        Help about any command
  init        Initialize a Cobra Application

Flags:
  -a, --author string    author name for copyright attribution (default "YOUR NAME")
      --config string    config file (default is $HOME/.cobra.yaml)
  -h, --help             help for cobra
  -l, --license string   name of license for the project
      --viper            use Viper for configuration (default true)

Use "cobra [command] --help" for more information about a command.

3.2 生成的代码框架

通过init生成代码，以下为主要生成的代码架构

#通过init生成包名为myapp的go程序框架
$ cobra init --pkg-name=myapp
Your Cobra applicaton is ready at /root/go/src/cobratest
#查看其目录结构，主要包含main.go及次级目录cmd
$ ls /root/go/src/cobratest 
-rw-r--r--  1 root  root  11358 12 11 11:09 LICENSE
drwxr-x--x  3 root  root  96 12 11 11:09 cmd
-rw-r--r--  1 root  root  641 12 11 11:09 main.go
#cmd下生成了root.go文件
$ ls -l /root/go/src/cobratest/cmd/
-rw-r--r--  1 root  root  2818 12 11 11:09 root.go

通过add添加子程序，即会在cmd下生成play.go文件

$ cobra add play
play created at /root/go/src/cobratest

$ ls -l cmd/
-rw-r--r--  1 root  root  1588 12 11 11:14 play.go
-rw-r--r--  1 root  root  2818 12 11 11:09 root.go

3.3 程序入口及调用逻辑

首先，我们看看main.go

package main

import "myapp/cmd"

#生成了main函数，并调用了cmd.Execute()，即cmd目录下root.go里的Execute()
func main() {
  cmd.Execute()
}

接下来我们看看cmd/root.go，可以看到root.go就做了几件事：

• 定义cobra.command结构体，关键是被注释掉的Run函数变量，该函数将承载整个程序的核心逻辑，否则该Run函数变量默认为空，该程序将只是cobra生成的架构，仅为空壳程序
• 定义Execute()函数，作为程序点火按钮，通过cobra框架内部流程，最终触发我们我们定义Run函数
• 定义init()和initConfig()用于做一些变量、配置的初始化

#root.go文件
package cmd

import (
  "fmt"
  "os"
  "github.com/spf13/cobra"

  homedir "github.com/mitchellh/go-homedir"
  "github.com/spf13/viper"

)

var cfgFile string

// rootCmd represents the base command when called without any subcommands
#构建cobra.command，Use即程序命令，Short、Long即程序功能说明，最后被注释的Run函数变量是核心关键，即程序主逻辑执行函数，需要我们自定义，否则为空
var rootCmd = &cobra.Command{
  Use:   "myapp",
  Short: "A brief description of your application",
  Long: `A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.`,
  // Uncomment the following line if your bare application
  // has an action associated with it:
  #Run生成时，默认为空，可以把注释去掉，即为空函数，一般我们会把业务程序主逻辑包装进该函数来执行，该函数定义好后，将会被后续Execute()触发执行
  //    Run: func(cmd *cobra.Command, args []string) { },
}

// Execute adds all child commands to the root command and sets flags appropriately.
// This is called by main.main(). It only needs to happen once to the rootCmd.
#Execute()即整个程序启动的点火按钮，如果将该程序比作一台车，前面的流程是将车做好，把准备工作都做到位，此处为将车开动起来的启动按钮
func Execute() {
  #通过执行cobra.Command内置的方法Execute()，然后一步步引导到执行上面定义好的Run函数，如果Run为空，则退出，此处我们就不做深入分析，如果有兴趣深挖的同学，可以网上查资料自行学习
  if err := rootCmd.Execute(); err != nil {
    fmt.Println(err)
    os.Exit(1)
  }
}

#初始化流程，主要为参数加载，参数来源一般命令行flag、配置文件、环境变量
func init() {
  cobra.OnInitialize(initConfig)

  // Here you will define your flags and configuration settings.
  // Cobra supports persistent flags, which, if defined here,
  // will be global for your application.

  rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.myapp.yaml)")


  // Cobra also supports local flags, which will only run
  // when this action is called directly.
  rootCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}


// initConfig reads in config file and ENV variables if set.
#通过配置文件、环境变量等方式解析、加载参数
func initConfig() {
  if cfgFile != "" {
    // Use config file from the flag.
    viper.SetConfigFile(cfgFile)
  } else {
    // Find home directory.
    home, err := homedir.Dir()
    if err != nil {
      fmt.Println(err)
      os.Exit(1)
    }

    // Search config in home directory with name ".myapp" (without extension).
    viper.AddConfigPath(home)
    viper.SetConfigName(".myapp")
  }

  viper.AutomaticEnv() // read in environment variables that match

  // If a config file is found, read it in.
  if err := viper.ReadInConfig(); err == nil {
    fmt.Println("Using config file:", viper.ConfigFileUsed())
  }
}

下面我们再来看看play.go文件，play.go一般为旁路逻辑，即次级命令或分支程序，代码流程和上面root.go类似

#play.go文件
package cmd

import (
    "fmt"
    "github.com/spf13/cobra"
)

// playCmd represents the play command
#同样构建一个cobra.Command，同样生成Run函数
var playCmd = &cobra.Command{
    Use:   "play",
    Short: "A brief description of your command",
    Long: `A longer description that spans multiple lines and likely contains examples
and usage of using your command. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.`,
    #Run函数默认为空逻辑，仅有个打印
    Run: func(cmd *cobra.Command, args []string) {
        fmt.Println("play called")
    },
}

#此处init()逻辑核心为rootCmd.AddCommand(playCmd)，主要把当前命令，添加进rootCmd，即上面root.go定义的，成为其二次命令使用，后续我们将该代码编译运行后，即可直观看到其效果
func init() {
    rootCmd.AddCommand(playCmd)

    // Here you will define your flags and configuration settings.

    // Cobra supports Persistent Flags which will work for this command
    // and all subcommands, e.g.:
    // playCmd.PersistentFlags().String("foo", "", "A help for foo")

    // Cobra supports local flags which will only run when this command
    // is called directly, e.g.:
    // playCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}

编译执行该程序，看其使用方法即效果

$ go build .
$ go build .

3.4 Run函数解析

我们先看看run函数如何启动的，然后通过kubelet的Run函数，剖析一下run函数的功能
首先，我们先看看Run函数定义

    #从注释看，该函数为实际工作的函数，绝大多数命令都只执行它
    // Run: Typically the actual work function. Most commands will only implement this.
    Run func(cmd *Command, args []string)

然后，进入Execute()里的rootCmd.Execute()，

#Execute()入口函数
func Execute() {
  if err := rootCmd.Execute(); err != nil {
    fmt.Println(err)
    os.Exit(1)
  }
}

#进入rootCmd.Execute()，实际执行rootCmd.ExecuteC()
func (c *Command) Execute() error {
    _, err := c.ExecuteC()
    return err
}

#我们再进入ExecuteC()，此处主要对主命令做校验及解析，然后根据解析的子命令或参数，进入下一轮execute()
func (c *Command) ExecuteC() (cmd *Command, err error) {
    ... #此处缩减了部分代码

    var flags []string
    if c.TraverseChildren {
           # 如果有子命令，根据传入参数遍历到对应子命令，并解析出对应参数
        cmd, flags, err = c.Traverse(args)
    } else {
        cmd, flags, err = c.Find(args)
    }
    ... #此处缩减了部分代码

    #我们可以看到，还有一层execute函数，以小写开头的内部函数，根据上面遍历的子命令与flag进行执行
    err = cmd.execute(flags)
    if err != nil {
        // Always show help if requested, even if SilenceErrors is in
        // effect
        if err == flag.ErrHelp {
            cmd.HelpFunc()(cmd, args)
            return cmd, nil
        }

        // If root command has SilenceErrors flagged,
        // all subcommands should respect it
        if !cmd.SilenceErrors && !c.SilenceErrors {
            c.PrintErrln("Error:", err.Error())
        }

        // If root command has SilenceUsage flagged,
        // all subcommands should respect it
        if !cmd.SilenceUsage && !c.SilenceUsage {
            c.Println(cmd.UsageString())
        }
    }
    return cmd, err
}

我们再进入下一轮execute()

func (c *Command) execute(a []string) (err error) {
    ... #此处缩减了部分代码

    # 校验命令、参数是否可执行，若不可，则返回错误
    if !c.Runnable() {
        return flag.ErrHelp
    }
    # cobra.Command结构体围绕Run共定义了10类执行体，分别是PersistentPreRun、PersistentPreRunE、PreRun、PreRunE、Run、RunE、PostRun、PostRunE、PersistentPostRun、PersistentPostRunE，这类函数Run和RunE的差别为是否返回错误信息，另外，他们的执行先后顺序依次是PersistentPreRun() -> PreRun() -> Run() -> PostRun() -> PersistentPostRun()，此处不对每个函数做详解，若有兴趣，可自行查资料或看分析cobra源码
     # 执行前置任务
    c.preRun()

    argWoFlags := c.Flags().Args()
    if c.DisableFlagParsing {
        argWoFlags = a
    }

    if err := c.ValidateArgs(argWoFlags); err != nil {
        return err
    }

     # 执行前置任务
    for p := c; p != nil; p = p.Parent() {
        if p.PersistentPreRunE != nil {
            if err := p.PersistentPreRunE(c, argWoFlags); err != nil {
                return err
            }
            break
        } else if p.PersistentPreRun != nil {
            p.PersistentPreRun(c, argWoFlags)
            break
        }
    }
    if c.PreRunE != nil {
        if err := c.PreRunE(c, argWoFlags); err != nil {
            return err
        }
    } else if c.PreRun != nil {
        c.PreRun(c, argWoFlags)
    }

    if err := c.validateRequiredFlags(); err != nil {
        return err
    }

     # 此处为核心，判断用户使用的是RunE还是Run，两者功能类似RunE会返回err信息，此处即进入到用户定义的核心Run函数体中了
    if c.RunE != nil {
        if err := c.RunE(c, argWoFlags); err != nil {
            return err
        }
    } else {
        c.Run(c, argWoFlags)
    }

     # 执行垫后任务
    if c.PostRunE != nil {
        if err := c.PostRunE(c, argWoFlags); err != nil {
            return err
        }
    } else if c.PostRun != nil {
        c.PostRun(c, argWoFlags)
    }
    for p := c; p != nil; p = p.Parent() {
        if p.PersistentPostRunE != nil {
            if err := p.PersistentPostRunE(c, argWoFlags); err != nil {
                return err
            }
            break
        } else if p.PersistentPostRun != nil {
            p.PersistentPostRun(c, argWoFlags)
            break
        }
    }

    return nil
}

4. kubelet Run函数解析

kubelet Run函数定义的具体代码，如下：

        #Run函数主体
        Run: func(cmd *cobra.Command, args []string) {
            // initial flag parse, since we disable cobra's flag parsing
            #解析参数
            if err := cleanFlagSet.Parse(args); err != nil {
                cmd.Usage()
                klog.Fatal(err)
            }

            // check if there are non-flag arguments in the command line
            #检查是否有命令未传入参数，若是，则返回错误和使用指引
            cmds := cleanFlagSet.Args()
            if len(cmds) > 0 {
                cmd.Usage()
                klog.Fatalf("unknown command: %s", cmds[0])
            }

            // short-circuit on help
            #检查是否设置了帮助指引
            help, err := cleanFlagSet.GetBool("help")
            if err != nil {
                klog.Fatal(`"help" flag is non-bool, programmer error, please correct`)
            }
            if help {
                cmd.Help()
                return
            }

            // short-circuit on verflag
            #若是查看版本信息，则返回版本信息及退出
            verflag.PrintAndExitIfRequested()
            #若是查看参数使用，则输出所有参数说明
            cliflag.PrintFlags(cleanFlagSet)

            // set feature gates from initial flags-based config
            #通过flag参数列表设置启动的feature gates
            if err := utilfeature.DefaultMutableFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {
                klog.Fatal(err)
            }

            // validate the initial KubeletFlags
            #校验kubelet flag相关参数传入是否合法，如果合法则初始化到对应变量
            if err := options.ValidateKubeletFlags(kubeletFlags); err != nil {
                klog.Fatal(err)
            }

            #检测ContainerRuntime 使用，如果设置成remote，则--pod-infra-container-image将被忽略
            if kubeletFlags.ContainerRuntime == "remote" && cleanFlagSet.Changed("pod-infra-container-image") {
                klog.Warning("Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead")
            }

            // load kubelet config file, if provided
            #检测并解析kubelet config文件
            if configFile := kubeletFlags.KubeletConfigFile; len(configFile) > 0 {
                kubeletConfig, err = loadConfigFile(configFile)
                if err != nil {
                    klog.Fatal(err)
                }
                // We must enforce flag precedence by re-parsing the command line into the new object.
                // This is necessary to preserve backwards-compatibility across binary upgrades.
                // See issue #56171 for more details.
                if err := kubeletConfigFlagPrecedence(kubeletConfig, args); err != nil {
                    klog.Fatal(err)
                }
                // update feature gates based on new config
                if err := utilfeature.DefaultMutableFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {
                    klog.Fatal(err)
                }
            }

            // We always validate the local configuration (command line + config file).
            // This is the default "last-known-good" config for dynamic config, and must always remain valid.
            #校验kubelet config文件中的配置内容是否合法，遇错误则退出
            if err := kubeletconfigvalidation.ValidateKubeletConfiguration(kubeletConfig); err != nil {
                klog.Fatal(err)
            }

            #检测kubelet congfig中cgroup相关配置内容是否合法
            if (kubeletConfig.KubeletCgroups != "" && kubeletConfig.KubeReservedCgroup != "") && (0 != strings.Index(kubeletConfig.KubeletCgroups, kubeletConfig.KubeReservedCgroup)) {
                klog.Warning("unsupported configuration:KubeletCgroups is not within KubeReservedCgroup")
            }

            // use dynamic kubelet config, if enabled
            #检测是否启用了kubelet热加载配置模式，及动态kubelet config，如果启动了，则按动态kubelet config模式启动kubelet
            var kubeletConfigController *dynamickubeletconfig.Controller
            if dynamicConfigDir := kubeletFlags.DynamicConfigDir.Value(); len(dynamicConfigDir) > 0 {
                var dynamicKubeletConfig *kubeletconfiginternal.KubeletConfiguration
                #获取kubelet动态配置信息
                dynamicKubeletConfig, kubeletConfigController, err = BootstrapKubeletConfigController(dynamicConfigDir,
                    func(kc *kubeletconfiginternal.KubeletConfiguration) error {
                        // Here, we enforce flag precedence inside the controller, prior to the controller's validation sequence,
                        // so that we get a complete validation at the same point where we can decide to reject dynamic config.
                        // This fixes the flag-precedence component of issue #63305.
                        // See issue #56171 for general details on flag precedence.
                        return kubeletConfigFlagPrecedence(kc, args)
                    })
                #获取失败，报错退出
                if err != nil {
                    klog.Fatal(err)
                }
                // If we should just use our existing, local config, the controller will return a nil config
                #获取到动态配置，则加载进来，否则报错退出
                if dynamicKubeletConfig != nil {
                    kubeletConfig = dynamicKubeletConfig
                    // Note: flag precedence was already enforced in the controller, prior to validation,
                    // by our above transform function. Now we simply update feature gates from the new config.
                    if err := utilfeature.DefaultMutableFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {
                        klog.Fatal(err)
                    }
                }
            }

            #通过kubelet flag和config构建kubelet server，用以支持对kubelet的请求
            // construct a KubeletServer from kubeletFlags and kubeletConfig
            kubeletServer := &options.KubeletServer{
                KubeletFlags:         *kubeletFlags,
                KubeletConfiguration: *kubeletConfig,
            }

            #通过kubelet server和DefaultFeatureGate，构建kubeletDeps，kubeletDeps主要集成了kubelet的运行依赖如：认证信息、第三方云厂商信息、事件记录、挂载、网络/卷插件、OOM管理等依赖组件调用链操作句柄
            // use kubeletServer to construct the default KubeletDeps
            kubeletDeps, err := UnsecuredDependencies(kubeletServer, utilfeature.DefaultFeatureGate)
            if err != nil {
                klog.Fatal(err)
            }

            #把kubelet config controller注入kubeletDeps
            // add the kubelet config controller to kubeletDeps
            kubeletDeps.KubeletConfigController = kubeletConfigController

            #设置信号上下文，将被kubelet和docker shim重复使用，做全局控制
            // set up signal context here in order to be reused by kubelet and docker shim
            ctx := genericapiserver.SetupSignalContext()

            #启动kubelet，把ctx、kubeletServer、kubeletDeps、utilfeature.DefaultFeatureGate这些准备好的参数传入，执行内层核心Run函数，此处暂不做深入剖析，后续我们将在kubelet源码解析第3章展开聊！
            // run the kubelet
            klog.V(5).Infof("KubeletConfiguration: %#v", kubeletServer.KubeletConfiguration)
            if err := Run(ctx, kubeletServer, kubeletDeps, utilfeature.DefaultFeatureGate); err != nil {
                klog.Fatal(err)
            }
        },

从上面流程，我们可以看到，kubelet的第一层Run函数，主要工作就是做些kubelet运行的前期准备，如：参数解析、校验、构建依赖、构建 server等，最后通过这些备好的参数运行kubelet，进入下一轮Run函数，然后真正开启kubelet的工作之旅