novaboot代码流程分析(五)：VM启动从neutron-dhcp-agent获取IP与MAC

1.network和subnet创建代码流程

这里，我们主要分析上面两个命令的代码流程，我们关注的重点是neutron-server如何下发的命令到neutron-dhcp-agent去创建相应的network的代码流程。

首先neutronclient发送HTTP请求给neutron-server，去执行下面的create函数。

#/neutron/api/v2/base.py:Controller
    def create(self, request, body=None, **kwargs):
        """Creates a new instance of the requested entity."""
        parent_id = kwargs.get(self._parent_id_name)
        self._notifier.info(request.context,
                            self._resource + '.create.start',
                            body)
        body = Controller.prepare_request_body(request.context, body, True,
                                               self._resource, self._attr_info,
                                               allow_bulk=self._allow_bulk)
        action = self._plugin_handlers[self.CREATE]
        ... ... ...
        def notify(create_result):
            notifier_method = self._resource + '.create.end'
            self._notifier.info(request.context,
                                notifier_method,
                                create_result)
            self._send_dhcp_notification(request.context,
                                         create_result,
                                         notifier_method)
            return create_result

        kwargs = {self._parent_id_name: parent_id} if parent_id else {}
        if self._collection in body and self._native_bulk:
            # plugin does atomic bulk create operations
            obj_creator = getattr(self._plugin, "%s_bulk" % action)
            objs = obj_creator(request.context, body, **kwargs)
            # Use first element of list to discriminate attributes which
            # should be removed because of authZ policies
            fields_to_strip = self._exclude_attributes_by_policy(
                request.context, objs[0])
            return notify({self._collection: [self._filter_attributes(
                request.context, obj, fields_to_strip=fields_to_strip)
                for obj in objs]})
        else:
            obj_creator = getattr(self._plugin, action)
            if self._collection in body:
                # Emulate atomic bulk behavior
                objs = self._emulate_bulk_create(obj_creator, request,
                                                 body, parent_id)
                return notify({self._collection: objs})
            else:
                kwargs.update({self._resource: body})
                obj = obj_creator(request.context, **kwargs)
                self._send_nova_notification(action, {},
                                             {self._resource: obj})
                return notify({self._resource: self._view(request.context,
                                                          obj)})

这个create函数实现的功能为：

1. 通过调用core_plugin的core resource所对应的action方法(如create_network)在数据库中保存resource信息。

2. 在保存自身的信息到数据库中后，通知agent去做相应的操作。

其实对于port，network，subnet的create操作都会走到这个create函数。这里我们主要分析创建network(包括subnet)时与neutron-dhcp-agent之间的操作。

在network(或subnet)的信息保存到数据库之后，通过notify函数(create函数内部的函数)通知neutron-dhcp-agent做相应的操作。

network和subnet发送的给neutron-dhcp-agent的方法(notifier_method)分别为network.create.end和subnet.create.end

#/neutron/api/v2/base.py:Controller
    def _send_dhcp_notification(self, context, data, methodname):
        if cfg.CONF.dhcp_agent_notification:
            if self._collection in data:
                for body in data[self._collection]:
                    item = {self._resource: body}
                    self._dhcp_agent_notifier.notify(context, item, methodname)
            else:
                self._dhcp_agent_notifier.notify(context, data, methodname)

dhcp_agent_notification为/etc/neuton/neutron.conf配置文件中的参数。

#/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py:DhcpAgentNotifyAPI
    def notify(self, context, data, method_name):
        # data is {'key' : 'value'} with only one key
        if method_name not in self.VALID_METHOD_NAMES:
            return
        obj_type = data.keys()[0]
        if obj_type not in self.VALID_RESOURCES:
            return
        obj_value = data[obj_type]
        network_id = None
        if obj_type == 'network' and 'id' in obj_value:
            network_id = obj_value['id']
        elif obj_type in ['port', 'subnet'] and 'network_id' in obj_value:
            network_id = obj_value['network_id']
        if not network_id:
            return
        method_name = method_name.replace(".", "_")
        if method_name.endswith("_delete_end"):
            if 'id' in obj_value:
                self._notify_agents(context, method_name,
                                    {obj_type + '_id': obj_value['id']},
                                    network_id)
        else:
            self._notify_agents(context, method_name, data, network_id)

#/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py:DhcpAgentNotifyAPI
    def _notify_agents(self, context, method, payload, network_id):
        """Notify all the agents that are hosting the network."""
        # fanout is required as we do not know who is "listening"
        no_agents = not utils.is_extension_supported(
            self.plugin, constants.DHCP_AGENT_SCHEDULER_EXT_ALIAS)
        fanout_required = method == 'network_delete_end' or no_agents

        # we do nothing on network creation because we want to give the
        # admin the chance to associate an agent to the network manually
        cast_required = method != 'network_create_end'

        if fanout_required:
            self._fanout_message(context, method, payload)
        elif cast_required:
            admin_ctx = (context if context.is_admin else context.elevated())
            network = self.plugin.get_network(admin_ctx, network_id)
            agents = self.plugin.get_dhcp_agents_hosting_networks(
                context, [network_id])

            # schedule the network first, if needed
            schedule_required = (
                method == 'subnet_create_end' or
                method == 'port_create_end' and
                not self._is_reserved_dhcp_port(payload['port']))
            if schedule_required:
                agents = self._schedule_network(admin_ctx, network, agents)

            enabled_agents = self._get_enabled_agents(
                context, network, agents, method, payload)
            for agent in enabled_agents:
                self._cast_message(
                    context, method, payload, agent.host, agent.topic)

在_notify_agents函数可以看出，由于创建network时，fanout_required和cast_required的值为False，所以创建network时，并不会通过neutron-dhcp-agent做什么操作。而创建port或subnet时，则会通过rpc方式执行neutron-dhcp-agent的方法，从而做一些操作。
[page]
所以下面我们主要分析在创建subnet时，neutron-dhcp-agent做了哪些操作。

#/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py:DhcpAgentNotifyAPI
    def _cast_message(self, context, method, payload, host,
                      topic=topics.DHCP_AGENT):
        """Cast the payload to the dhcp agent running on the host."""
        cctxt = self.client.prepare(topic=topic, server=host)
        cctxt.cast(context, method, payload=payload)

#/neutron/agent/dhcp/agent.py:DhcpAgent
    @utils.synchronized('dhcp-agent')
    def subnet_update_end(self, context, payload):
        """Handle the subnet.update.end notification event."""
        network_id = payload['subnet']['network_id']
        self.refresh_dhcp_helper(network_id)

    # Use the update handler for the subnet create event.
    subnet_create_end = subnet_update_end

看到/neutron/agent/dhcp/agent.py:DhcpAgent类是不是很熟悉，这正是上一篇文章中分析的在neutron-dhcp-agent服务启动时，创建的DhcpAgent对象。

#/neutron/agent/dhcp/agent.py:DhcpAgent
    def refresh_dhcp_helper(self, network_id):
        """Refresh or disable DHCP for a network depending on the current state
        of the network.
        """
        old_network = self.cache.get_network_by_id(network_id)
        if not old_network:
            # DHCP current not running for network.
            return self.enable_dhcp_helper(network_id)

        network = self.safe_get_network_info(network_id)
        if not network:
            return

        old_cidrs = set(s.cidr for s in old_network.subnets if s.enable_dhcp)
        new_cidrs = set(s.cidr for s in network.subnets if s.enable_dhcp)

        if new_cidrs and old_cidrs == new_cidrs:
            self.call_driver('reload_allocations', network)
            self.cache.put(network)
        elif new_cidrs:
            if self.call_driver('restart', network):
                self.cache.put(network)
        else:
            self.disable_dhcp_helper(network.id)

refresh_dhcp_helper函数将对比self.cache中存放的network信息与新创建的network信息，通过对比结果做相应操作。

1. 如果self.cache中没有新创建的network信息，则调用enable_dhcp_helper函数，该函数最终调用driver(dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq)的enable函数。

2. 如果self.cache中有新创建的network信息，则判断old network与new network的cidr是否相等，根据结果判断调用driver中的reload_allocations函数还是restart函数，又或者是disable函数。

对于执行neutron.agent.linux.dhcp.Dnsmasq driver中的相关函数(enable, reload_allocations, restart,disable)，我们在这里本篇文章就不分析了，具体查看《neutron-dhcp-agent服务启动流程》。

PS：在《neutron-dhcp-agent服务启动流程》文章中，我们也有分析，如果创建的subnet并没有enable dhcp，则neutron-dhcp-agent不会为该network创建namespace，创建device以及创建dnsmasq进程。具体代码如下。

#/neutron/agent/dhcp/agent.py:DhcpAgent
    def configure_dhcp_for_network(self, network):
        if not network.admin_state_up:
            return

        enable_metadata = self.dhcp_driver_cls.should_enable_metadata(
                self.conf, network)
        dhcp_network_enabled = False

        for subnet in network.subnets:
            if subnet.enable_dhcp:
                if self.call_driver('enable', network):
                    dhcp_network_enabled = True
                    self.cache.put(network)
                break

        if enable_metadata and dhcp_network_enabled:
            for subnet in network.subnets:
                if subnet.ip_version == 4 and subnet.enable_dhcp:
                    self.enable_isolated_metadata_proxy(network)
                    break

小结一下：

1. 创建network，subnet和port时，首先调用core_plugin提供的函数向数据库写入自身的信息。

2. 写完数据库之后，subnet和port通过rpc方式，执行neutron-dhcp-agent启动时创建的DhcpAgent对象提供的函数(这里创建network时，并不会执行neutron-dhcp-agent所提供的函数方法)。

3. neutron-dhcp-agent对创建subnet的操作可以参考《neutron-dhcp-agent服务启动流程》文章。

对于neutron-dhcp-agent对创建VM时所创建的port的操作我们在下一小节分析。

2. VM启动时从neutron-dhcp-agent获取IP与MAC的代码流程分析

2.1 更新dnsmasq配置文件信息

接到《nova boot代码流程分析(三)：nova与neutron的plugin交互》的创建port的代码流程分析，因为在那篇文章中，主要关注的如何将创建的IP和MAC保存到数据库，并未关注VM实际如何从neutron-dhcp-agent获取IP和MAC的代码流程，所以在这篇文章中，我们将关注VM实际获取IP和MAC的代码流程。

#/nova/network/neutronv2/api.py:API
    def _create_port(self, port_client, instance, network_id, port_req_body,
                     fixed_ip=None, security_group_ids=None,
                     available_macs=None, dhcp_opts=None):
        """Attempts to create a port for the instance on the given network.

        :param port_client: The client to use to create the port.
        :param instance: Create the port for the given instance.
        :param network_id: Create the port on the given network.
        :param port_req_body: Pre-populated port request. Should have the
            device_id, device_owner, and any required neutron extension values.
        :param fixed_ip: Optional fixed IP to use from the given network.
        :param security_group_ids: Optional list of security group IDs to
            apply to the port.
        :param available_macs: Optional set of available MAC addresses,
            from which one will be used at random.
        :param dhcp_opts: Optional DHCP options.
        :returns: ID of the created port.
        :raises PortLimitExceeded: If neutron fails with an OverQuota error.
        :raises NoMoreFixedIps: If neutron fails with
            IpAddressGenerationFailure error.
        """
        try:
            if fixed_ip:
                port_req_body['port']['fixed_ips'] = [
                    {'ip_address': str(fixed_ip)}]
            port_req_body['port']['network_id'] = network_id
            port_req_body['port']['admin_state_up'] = True
            port_req_body['port']['tenant_id'] = instance.project_id
            if security_group_ids:
                port_req_body['port']['security_groups'] = security_group_ids
            if available_macs is not None:
                if not available_macs:
                    raise exception.PortNotFree(
                        instance=instance.uuid)
                mac_address = available_macs.pop()
                port_req_body['port']['mac_address'] = mac_address
            if dhcp_opts is not None:
                port_req_body['port']['extra_dhcp_opts'] = dhcp_opts
            port_id = port_client.create_port(port_req_body)['port']['id']
            LOG.debug('Successfully created port: %s', port_id,
                      instance=instance)
            return port_id
        except neutron_client_exc.IpAddressInUseClient:
            LOG.warning(_LW('Neutron error: Fixed IP %s is '
                            'already in use.'), fixed_ip)
            msg = _("Fixed IP %s is already in use.") % fixed_ip
            raise exception.FixedIpAlreadyInUse(message=msg)
        except neutron_client_exc.OverQuotaClient:
            LOG.warning(_LW(
                'Neutron error: Port quota exceeded in tenant: %s'),
                port_req_body['port']['tenant_id'], instance=instance)
            raise exception.PortLimitExceeded()
        except neutron_client_exc.IpAddressGenerationFailureClient:
            LOG.warning(_LW('Neutron error: No more fixed IPs in network: %s'),
                        network_id, instance=instance)
            raise exception.NoMoreFixedIps(net=network_id)
        except neutron_client_exc.MacAddressInUseClient:
            LOG.warning(_LW('Neutron error: MAC address %(mac)s is already '
                            'in use on network %(network)s.') %
                        {'mac': mac_address, 'network': network_id},
                        instance=instance)
            raise exception.PortInUse(port_id=mac_address)
        except neutron_client_exc.NeutronClientException:
            with excutils.save_and_reraise_exception():
                LOG.exception(_LE('Neutron error creating port on network %s'),
                              network_id, instance=instance)

这部分代码是从《nova boot代码流程分析(三)：nova与neutron的plugin交互》文章中提取出来的代码，该代码是nova-compute通过HTTP请求在neutron中创建port信息。然后到达我们第一小节分析的create函数。

#/neutron/api/v2/base.py:Controller
    def create(self, request, body=None, **kwargs):
        """Creates a new instance of the requested entity."""
        parent_id = kwargs.get(self._parent_id_name)
        self._notifier.info(request.context,
                            self._resource + '.create.start',
                            body)
        body = Controller.prepare_request_body(request.context, body, True,
                                               self._resource, self._attr_info,
                                               allow_bulk=self._allow_bulk)
        action = self._plugin_handlers[self.CREATE]
        ... ... ...
        def notify(create_result):
            notifier_method = self._resource + '.create.end'
            self._notifier.info(request.context,
                                notifier_method,
                                create_result)
            self._send_dhcp_notification(request.context,
                                         create_result,
                                         notifier_method)
            return create_result

        kwargs = {self._parent_id_name: parent_id} if parent_id else {}
        if self._collection in body and self._native_bulk:
            # plugin does atomic bulk create operations
            obj_creator = getattr(self._plugin, "%s_bulk" % action)
            objs = obj_creator(request.context, body, **kwargs)
            # Use first element of list to discriminate attributes which
            # should be removed because of authZ policies
            fields_to_strip = self._exclude_attributes_by_policy(
                request.context, objs[0])
            return notify({self._collection: [self._filter_attributes(
                request.context, obj, fields_to_strip=fields_to_strip)
                for obj in objs]})
        else:
            obj_creator = getattr(self._plugin, action)
            if self._collection in body:
                # Emulate atomic bulk behavior
                objs = self._emulate_bulk_create(obj_creator, request,
                                                 body, parent_id)
                return notify({self._collection: objs})
            else:
                kwargs.update({self._resource: body})
                obj = obj_creator(request.context, **kwargs)
                self._send_nova_notification(action, {},
                                             {self._resource: obj})
                return notify({self._resource: self._view(request.context,
                                                          obj)})

create函数首先创建IP和MAC信息保存到neutron数据库中，具体参考《nova boot代码流程分析(三)：nova与neutron的plugin交互》文章。然后调用notify函数发送port.create.end给neutron-dhcp-agent。最终neutron-dhcp-agent将调用port_create_end函数。

#/neutron/agent/dhcp/agent.py:DhcpAgent
    @utils.synchronized('dhcp-agent')
    def port_update_end(self, context, payload):
        """Handle the port.update.end notification event."""
        updated_port = dhcp.DictModel(payload['port'])
        network = self.cache.get_network_by_id(updated_port.network_id)
        if network:
            driver_action = 'reload_allocations'
            if self._is_port_on_this_agent(updated_port):
                orig = self.cache.get_port_by_id(updated_port['id'])
                # assume IP change if not in cache
                old_ips = {i['ip_address'] for i in orig['fixed_ips'] or []}
                new_ips = {i['ip_address'] for i in updated_port['fixed_ips']}
                if old_ips != new_ips:
                    driver_action = 'restart'
            self.cache.put_port(updated_port)
            self.call_driver(driver_action, network)

    def _is_port_on_this_agent(self, port):
        thishost = utils.get_dhcp_agent_device_id(
            port['network_id'], self.conf.host)
        return port['device_id'] == thishost

    # Use the update handler for the port create event.
    port_create_end = port_update_end

具体如何到达该代码流程，可以参考第1小节创建subnet的代码流程。这里将port_update_end函数赋给port_create_end函数，所以最终是调用port_update_end函数。port_update_end函数将比较self.cache中保存的ip信息与最新数据库中相对应的network中对应的ip信息(从neutron-server传递下来的，即payload)。
[page]
正常流程下，因为我们创建VM会创建新的port信息到数据库中，而neutron-dhcp-agent的self.cache中还保存着上一次更新的network信息，所以导致数据库中port信息与self.cache中保存的port信息不一致，因此将更新self.cache中的port信息，且执行reload_allocations函数，重新更新dnsmasq的配置文件。

#/neutron/agent/linux/dhcp.py:Dnsmasq
    def reload_allocations(self):
        """Rebuild the dnsmasq config and signal the dnsmasq to reload."""

        # If all subnets turn off dhcp, kill the process.
        if not self._enable_dhcp():
            self.disable()
            LOG.debug('Killing dnsmasq for network since all subnets have '
                      'turned off DHCP: %s', self.network.id)
            return

        self._release_unused_leases()
        self._spawn_or_reload_process(reload_with_HUP=True)
        LOG.debug('Reloading allocations for network: %s', self.network.id)
        self.device_manager.update(self.network, self.interface_name)

调用_release_unused_leases函数release在leases文件(/var/lib/neutron/dhcp/43c0e274-28e3-482e-a32b-d783980fc3ed/leases)中未被使用的ip和mac。然后再重新加载dnsmasq进程所需的配置文件，由于加载所需的配置文件我们在《neutron-dhcp-agent服务启动流程》中有分析，所以我们这里主要分析release在leases文件中未被使用的ip和mac。

#/neutron/agent/dhcp/agent.py:DhcpAgent
    def _release_unused_leases(self):
        filename = self.get_conf_file_name('host')
        old_leases = self._read_hosts_file_leases(filename)

        new_leases = set()
        for port in self.network.ports:
            for alloc in port.fixed_ips:
                new_leases.add((alloc.ip_address, port.mac_address))

        for ip, mac in old_leases - new_leases:
            self._release_lease(mac, ip)

#/neutron/agent/dhcp/agent.py:DhcpAgent
    def _release_lease(self, mac_address, ip):
        """Release a DHCP lease."""
        cmd = ['dhcp_release', self.interface_name, ip, mac_address]
        ip_wrapper = ip_lib.IPWrapper(namespace=self.network.namespace)
        ip_wrapper.netns.execute(cmd, run_as_root=True)

_release_unused_leases函数读取host文件(/var/lib/neutron/dhcp/43c0e274-28e3-482e-a32b-d783980fc3ed/host)中的被dnsmasq分配的ip和mac信息，将这些信息与数据库中network的ip和mac信息作对比，如果dnsmasq分配的ip和mac信息并未在数据库中，说明此ip和mac信息未被使用，所以调用_release_lease函数对其进行释放。

_release_lease函数是通过dhcp_release命令向dnsmasq进程监听接口ns-xxx发送DHCPRELEASE请求来释放未被使用的ip和mac信息。即删除leases文件未被使用的ip和mac信息。下面是一个测试用例。

开始leases文件信息：

假设我们将release红色部分的ip和mac信息，则执行以下操作。

在/var/log/messages文件中，可以看到如下日志。

执行dhcp_release命令后的leases文件信息：

可以看出，ip为1.1.1.9相关的信息被删除了。

这是_release_unused_leases函数的功能，而reload_allocations函数剩下的代码则是重新配置dnsmasq的配置文件，假设我们创建VM在数据库中分配的ip为1.1.1.10，那么在host文件中将把该ip及相对应的mac写入进去，待VM启动且发送dhcp discover请求时将该IP和mac分配于它。

2.2 分配ip和mac给VM

在创建VM开始时，首先为VM在数据库中创建port信息，创建port信息完成后，通知neutron-dhcp-agent更新dnsmasq的配置文件(包括host和addn_hosts文件)信息。如下

这是在VM启动之前，就已经保存到dnsmasq的配置文件中的信息。

等待VM启动时，发送dhcp discover请求。

VM启动的有关dhcp的log如下(这是dhcp client端)。

/var/log/messages的有关dhcp的log如下(这是dhcp server端)。

同时neutron-dhcp-agent更新dnsmasq的leases文件信息。

Dhcp client与dhcp server的交互过程如下图所示。

这里创建VM时，从neutron-dhcp-agent获取ip和mac的代码流程便分析完成。总结一下：

1. 创建VM时，nova-compute与neutron的plugin交互，在neutron的数据库中创建VM所需的port信息。

2. neutron数据库中的port信息创建完成后，通知neutron-dhcp-agent去执行port_create_end函数。该函数将数据库中的port中的ip和mac信息加载到dnsmasq所需的配置文件中(包括host和addn_hosts文件)。

3. 在VM启动时，广播dhcp discover请求，当dnsmasq进程的监听接口ns-xxx监听到这种请求时，dnsmasq进程将根据配置文件(host和leases文件)中的内容去判定是否有未分配的ip和mac为请求者进行提供。

4. 最终VM便真实的获取到与保存在数据库中的ip和mac信息。neutron-dhcp-agent只是将所创建VM的ip和mac信息从数据库中获取到自己的配置文件中，然后等到VM启动时，为它提供。因此neutron-dhcp-agent相当于在VM和数据库之间起了个中间桥梁的作用。