django源码分析之项目创建

注:本文分析涉及到的源码基于Django stable/2.0.x 分支。

计算机大部分思想都是来自于现实生活,所以完全可以用日常生活积累的常识去理解计算机里面的概念。
比如开发一套完整的软件系统,就好比去开发一片商业区。开发商业区的第一步就是要有一块空地;软件系统也是,第一步需要有一个空的项目。在Django世界里,这很简单,用下面命令就可以创建Django空项目。

django-admin.py startproject demo_project

有了一块空地后,下一步就是在它上面建一栋栋大楼(以及给大楼起名)。而在Django中,与之对应的就是创建app的过程,同样非常简单的执行下面命令就可,其中demo_app1就是给大楼起的名字。

python manage.py startapp demo_app1

然而要弄好一片商业区必然没这么简单,还需要对功能不同的大楼进行不同的设计,建造和装修,以及道路指引牌,甚至还有和安全相关的一系列配套设施等等。
这些在后续的文章中会一一道来,这篇文章并不打算继续介绍它们,而是先深入分析上面两个看似非常简单的过程(startproject和startapp),因为本文并不想写成如何教人使用Django的说明书类的文章。
目前为止仅仅执行两个命令,就可以让Django帮你创建好了项目。下文将拨开云雾,来分析它们背后实际的代码逻辑。
好了,先晒源码,
django-admin.py源码:

#!/usr/bin/env python
from django.core import management

if __name__ == "__main__":
    management.execute_from_command_line()

manage.py源码:

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "demo_project.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)

文件django-admin.pymanage.py都用到了django.core.management模块,并执行execute_from_command_line方法。
此外后者多做了两点:1)设置环境变量DJANGO_SETTINGS_MODULE为当前项目的settings文件;2)判断Django是否能被正常导入使用。
上面的第一点不是本文的重点,以后的文章再来讨论(注意这里不讨论,并不是说DJANGO_SETTINGS_MODULE不重要,恰恰相反,它很重要,它是Django项目的入口,指向项目的配置文件,该配置文件中指向ROOT_URLCONFROOT_URLCONF指向视图以及其他的部分,Django项目需要定位到它们之后才能正常运行)。
继续看django.core.management.execute_from_command_line方法:

def execute_from_command_line(argv=None):
    """Run a ManagementUtility."""
    utility = ManagementUtility(argv)
    utility.execute()

其中execute_from_command_line方法只是实例化了类ManagementUtility,接下来的重点查看ManagementUtility.execute方法。
该方法有点长,为了方便我们拆开来分析(源码位置django/core/management/__init__.py):

def execute(self):
    """
    Given the command-line arguments, figure out which subcommand is being
    run, create a parser appropriate to that command, and run it.
    """
    try:
        subcommand = self.argv[1]
    except IndexError:
        subcommand = 'help'  # Display help if no arguments were given.

上面这段代码的目的在于获取命令参数self.argv[1];如果用户没有输入命令参数,将用help命令为默认的参数,对应本文开头的django-admin.py startproject demo_projectpython manage.py startapp demo_app1两条指令,系统获取到的命令参数为startprojectstartapp

    # Preprocess options to extract --settings and --pythonpath.
    # These options could affect the commands that are available, so they
    # must be processed early.
    parser = CommandParser(None, usage="%(prog)s subcommand [options] [args]", add_help=False)
    parser.add_argument('--settings')
    parser.add_argument('--pythonpath')
    parser.add_argument('args', nargs='*')  # catch-all
    try:
        options, args = parser.parse_known_args(self.argv[2:])
        handle_default_options(options)
    except CommandError:
        pass  # Ignore any option errors at this point.

接下来用CommandParser类来解析剩下的命令行参数(该类只是对类ArgumentParser的简单封装),这段代码主要是预处理settingspythonpath两个可选参数。解析这两个参数到options中(其他参数放在args),然后由方法handle_default_options去处理options

def handle_default_options(options):
    """
    Include any default options that all commands should accept here
    so that ManagementUtility can handle them before searching for
    user commands.
    """
    if options.settings:
        os.environ['DJANGO_SETTINGS_MODULE'] = options.settings
    if options.pythonpath:
        sys.path.insert(0, options.pythonpath)

从上面代码中可以看到handle_default_options()根据对象optionssettingspythonpath来设置环境变量和设置python模块的搜索路径。
继续分析ManagementUtility.execute函数:

    try:
        settings.INSTALLED_APPS
    except ImproperlyConfigured as exc:
        self.settings_exception = exc
    except ImportError as exc:
        self.settings_exception = exc

这段代码可能会让读者诧异,莫名出现了settings对象,其实在这个源代码文件开头有行代码from django.conf import settings引入了settings,在执行这行代码时,会读取os.environ中的DJANGO_SETTINGS_MODULE配置,加载项目配置文件后生成了settings对象(这里先简单的说明下,以后会有专门的文章讲配置文件加载过程)。
代码settings.INSTALLED_APPS是用来导入配置文件中所有app(通过查看文件django/conf/__init__.py,发现settings = LazySettings(),它具有__getattr__方法,所以这行代码相当于调用了LazySettings().__getattr__(INSTALLED_APPS),该方法获取属性时如果没有导入配置则导入)。

if settings.configured:
    # Start the auto-reloading dev server even if the code is broken.
    # The hardcoded condition is a code smell but we can't rely on a
    # flag on the command class because we haven't located it yet.
    if subcommand == 'runserver' and '--noreload' not in self.argv:
        try:
            autoreload.check_errors(django.setup)()
        except Exception:
            # The exception will be raised later in the child process
            # started by the autoreloader. Pretend it didn't happen by
            # loading an empty list of applications.
            apps.all_models = defaultdict(OrderedDict)
            apps.app_configs = OrderedDict()
            apps.apps_ready = apps.models_ready = apps.ready = True

            # Remove options not compatible with the built-in runserver
            # (e.g. options for the contrib.staticfiles' runserver).
            # Changes here require manually testing as described in
            # #27522.
            _parser = self.fetch_command('runserver').create_parser('django', 'runserver')
            _options, _args = _parser.parse_known_args(self.argv[2:])
            for _arg in _args:
                self.argv.remove(_arg)

    # In all other cases, django.setup() is required to succeed.
    else:
        django.setup()

第一行代码表示如果settings对象已经配置好,就会执行django.setup()方法(注:第一个分支专门针对runserver启动的服务,并且使用autoreload机制的时候做的特殊处理,autoreload机制也是挺有意思的地方,后续也会写专门的文章来说明,这里先简单略过,只需要知道其实第一个分支的autoreload.check_errors(django.setup)()也是执行了django.setup()即可)
接着看django.setup(),在文件django/__init__.py中:

from django.utils.version import get_version

VERSION = (2, 0, 5, 'alpha', 0)

__version__ = get_version(VERSION)


def setup(set_prefix=True):
    """
    Configure the settings (this happens as a side effect of accessing the
    first setting), configure logging and populate the app registry.
    Set the thread-local urlresolvers script prefix if `set_prefix` is True.
    """
    from django.apps import apps
    from django.conf import settings
    from django.urls import set_script_prefix
    from django.utils.log import configure_logging

    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
    if set_prefix:
        set_script_prefix(
            '/' if settings.FORCE_SCRIPT_NAME is None else settings.FORCE_SCRIPT_NAME
        )
    apps.populate(settings.INSTALLED_APPS)

django.setup()主要做了3件事:1)对logging模块进行了配置处理;2)设置当前线程的script_prefix,可以理解成当前目录的前缀;3)循环载入之前从配置文件中导入的apps和models:(通过from django.apps import apps去查看文件django/apps/registry.py看到apps = Apps(installed_apps=None),这里Apps初始化的时候定义了全局的多个字典、变量、线程锁等,然后通过apps.populate(settings.INSTALLED_APPS)方法循环载入之前从配置文件中导入的app和model,该方法是线程安全和幂等的,但不可重入,也等下篇文章再来详细讲解)。
继续ManagementUtility.execute函数:

    self.autocomplete()

上面这行代码主要为了提供命令自动补全功能。不详细讲了,但是需要知道Bash几个内置的变量,COMP_WORDS: 类型为数组,存放当前命令行中输入的所有单词;COMP_CWORD: 类型为整数,当前光标下输入的单词位于COMP_WORDS数组中的索引;COMPREPLY: 类型为数组,候选的补全结果。

    if subcommand == 'help':
        if '--commands' in args:
            sys.stdout.write(self.main_help_text(commands_only=True) + '\n')
        elif not options.args:
            sys.stdout.write(self.main_help_text() + '\n')
        else:
            self.fetch_command(options.args[0]).print_help(self.prog_name, options.args[0])
    # Special-cases: We want 'django-admin --version' and
    # 'django-admin --help' to work, for backwards compatibility.
    elif subcommand == 'version' or self.argv[1:] == ['--version']:
        sys.stdout.write(django.get_version() + '\n')
    elif self.argv[1:] in (['--help'], ['-h']):
        sys.stdout.write(self.main_help_text() + '\n')
    else:
        self.fetch_command(subcommand).run_from_argv(self.argv)

这段代码前几个分支,主要是处理help命令和version命令,最后一个分支self.fetch_command(subcommand).run_from_argv(self.argv)才是我们一开始讨论的那两个命令的真实入口。
其中fetch_command调用get_commands从下面几个目录找命令对象(命令类都继承自BaseCommand,并实现handle()方法)

  1. django/core/management/commands目录
  2. <INSTALLED_APPS>/management/commands/目录

根据返回的subcommand实例,执行run_from_argv()方法。
对应我们文章开始讨论的地方,也就是相当于调用了startproject.run_from_argv(self.argv)startapp.run_from_argv(self.argv),它们对应的源码如下
core/management/commands/startproject.py

from django.core.management.templates import TemplateCommand

from ..utils import get_random_secret_key


class Command(TemplateCommand):
    help = (
        "Creates a Django project directory structure for the given project "
        "name in the current directory or optionally in the given directory."
    )
    missing_args_message = "You must provide a project name."

    def handle(self, **options):
        project_name = options.pop('name')
        target = options.pop('directory')

        # Create a random SECRET_KEY to put it in the main settings.
        options['secret_key'] = get_random_secret_key()

        super().handle('project', project_name, target, **options)

core/management/commands/startapp.py

from django.core.management.templates import TemplateCommand


class Command(TemplateCommand):
    help = (
        "Creates a Django app directory structure for the given app name in "
        "the current directory or optionally in the given directory."
    )
    missing_args_message = "You must provide an application name."

    def handle(self, **options):
        app_name = options.pop('name')
        target = options.pop('directory')
        super().handle('app', app_name, target, **options)

这两个命令都继承了TemplateCommand类,而且它们的代码逻辑也几乎一样,只是传入的参数不同,所以接下来通过分析TemplateCommand.handle()即可得知真相。
细心的读者可能会发现,上面明明调用的是方法run_from_argv(),但是startprojectstartapp两命令都没有这个方法,其实方法run_from_argv()是它们继承的父类BaseCommand里的方法(具体可以查看源码django/core/management/base.py),run_from_argv()最后会调用它们的handle()方法。
现在已经距离真相越来越近了,继续回到TemplateCommand.handle()
handle方法代码也有点多,只挑关键的来分析

def handle(self, app_or_project, name, target=None, **options):
    ...
    for root, dirs, files in os.walk(template_dir):
        ...
        for filename in files:
            old_path = path.join(root, filename)
            new_path = path.join(top_dir, relative_dir,
                                 filename.replace(base_name, name))
            for old_suffix, new_suffix in self.rewrite_template_suffixes:
                if new_path.endswith(old_suffix):
                    new_path = new_path[:-len(old_suffix)] + new_suffix
                    break  # Only rewrite once
            ...
            # Only render the Python files, as we don't want to
            # accidentally render Django templates files
            if new_path.endswith(extensions) or filename in extra_files:
                with open(old_path, 'r', encoding='utf-8') as template_file:
                    content = template_file.read()
                template = Engine().from_string(content)
                content = template.render(context)
                with open(new_path, 'w', encoding='utf-8') as new_file:
                    new_file.write(content)
            else:
                shutil.copyfile(old_path, new_path)

            if self.verbosity >= 2:
                self.stdout.write("Creating %s\n" % new_path)
            try:
                shutil.copymode(old_path, new_path)
                self.make_writeable(new_path)
            except OSError:
                self.stderr.write(
                      "Notice: Couldn't set permission bits on %s. You're "
                      "probably using an uncommon filesystem setup. No "
                      "problem." % new_path, self.style.NOTICE)
        ...

核心逻辑是遍历处理template_dir目录下的文件,这里的template_dirstartproject命令和startapp命令中分别对应目录django/conf/project_template/和目录django/conf/app_template/。目录结构如下

模版目录

为了更好的理解,先把文章一开始执行startprojectstartapp命令生成的目录结构也拿出来,对比后发现几乎完全一样,除了文件后缀名不一样:
project目录

app目录

其实到这里为止,已经能猜到大致过程了,在执行startproject的时候,会遍历django/conf/project_template/下面的文件,把源文件的后缀.py-tpl改成.py,然后根据设置好的模版引擎生成相应的文件。这样项目就创建完成了。同样,startapp也是一样的道理。
下面几行代码是专门改文件后缀的

for old_suffix, new_suffix in self.rewrite_template_suffixes:
    if new_path.endswith(old_suffix):
        new_path = new_path[:-len(old_suffix)] + new_suffix
        break  # Only rewrite once

其中self.rewrite_template_suffixes就是个python 元组对象,里面包含了原后缀名(py-tpl),以及新后缀名(py)

# Rewrite the following suffixes when determining the target filename.
rewrite_template_suffixes = (
    # Allow shipping invalid .py files without byte-compilation.
    ('.py-tpl', '.py'),
)

现在还剩下模版引擎这块,照样先晒代码

# Only render the Python files, as we don't want to
# accidentally render Django templates files
for filename in files:
    if new_path.endswith(extensions) or filename in extra_files:
        with open(old_path, 'r', encoding='utf-8') as template_file:
            content = template_file.read()
        template = Engine().from_string(content)
        content = template.render(context)
        with open(new_path, 'w', encoding='utf-8') as new_file:
            new_file.write(content)
    else:
        shutil.copyfile(old_path, new_path)

这段代码主要是读取模版文件里面的内容,通过调用方法Engine().from_string()生成Template模版对象。
比如下面settings.py-tpl模版文件内容(只截取了部分)被用来生成Template模版对象

"""
Django settings for {{ project_name }} project.

Generated by 'django-admin startproject' using Django {{ django_version }}.

For more information on this file, see
https://docs.djangoproject.com/en/{{ docs_version }}/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/{{ docs_version }}/ref/settings/
"""

import os

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/{{ docs_version }}/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '{{ secret_key }}'
...

接着template.render()会根据context内容渲染模版,context内容如下

context = Context({
    **options,
    base_name: name,
    base_directory: top_dir,
    camel_case_name: camel_case_value,
    'docs_version': get_docs_version(),
    'django_version': django.__version__,
}, autoescape=False)

可见,context包含的docs_versiondjango_versionoptions里面的内容会用来替换模版文件里被{{}}包含的内容,比如{{ docs_version }},然后替换后的新内容会被写入到对应的新文件里,这样Django就帮忙生成了项目需要的所有默认文件。
到现在为止,我们已经知道了文章开头那两个简单命令具体做了什么,可以让我们轻松的完成项目的创建。
最后总结:
startprojectstartapp属于初期创建项目阶段的命令,所以更多的是完成项目配置相关的工作,之后再根据用户输入的参数subcommand到命令工具集中去找到对应的命令对象。继承自TemplateCommand类的命令对象,使用模版引擎把相应template_dir下面的模版文件渲染生成新项目需要的文件和目录,最终Django就帮我们创建好了新项目。

推荐阅读更多精彩内容