azeqjz OpenStack： Neutron学习与实践
Neutron学习
例子：虚拟机使用VXLAN内部网络，通过VLAN类型的浮动IP从控制节点访问。
网络流向图
| 设备类型 | 设备 | 备注 | 分块 | |
|---|---|---|---|---|
| 虚拟机 | instance-1 | A | ||
| 虚拟机网口 | tap | 虚拟机侧接口 | A | 无法配置ACL规则 |
| linux虚拟接口 | vnet | 网桥接口 | B | |
| Linux Bridge | qbr | 网桥 | B | 安全组策略实现 |
| veth pair-A | qvb | 网桥接口 | B | quantum veth bridge |
| veth pair-B | qvo | 网桥接口 | C | quantum veth ovs |
| OVS Bridge | br-int | 网桥 | C | |
| veth pair-A | int-br-tun/ethx | 网桥接口namespace(route) | C | 内部vlan与外部vlan转换 |
| veth pair-B | phy-br-tun/ethx | 网桥接口namespace(route) | D | 外部vlan与内部vlan转换 |
| OVS Bridge | br-tun/eth | 网桥 | D | |
| 外部网络网口 | eth1 | 物理接口 | D | |
| 外部网络 | ext_net | D |
br-int br-integration OpenStack内部通信。
br-tun/eth* br-eth应该是包含在br-trunk内，放通多个vlan，转换为外部VLAN。
br-ex正常转发。
veth pair相当于虚拟网线。
namespace有两种：dhcp/route。
[student@workstation ~(developer1-research)]$ openstack server list
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
| ID | Name | Status | Networks | Image Name |
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
| 637228e8-2d9c-43d8-9f1f-354930139745 | research-web1 | ACTIVE | research-network1=192.168.1.9, 172.25.250.109 | rhel7 |
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
网络信息
[heat-admin@overcloud-controller-0 ~]$ openstack network list
+--------------------------------------+---------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------------------+--------------------------------------+
| 02471475-7999-4bf7-8b9d-2426e0cd83da | finance-network1 | c088be6d-8169-4e40-9c33-9e367ba1b7fe |
| 2ad72164-1f03-4678-a953-855da36750f4 | production-network1 | b7ee5c93-ae8a-453c-aa49-8b65bf6826ed |
| 33efb2c7-8c95-4d11-8662-405511021490 | provider-172.25.250 | b2442408-7390-4e08-906e-e99654318034 |
| d7b2a035-78d3-4525-a2e9-0841bbb09086 | research-network1 | 75f97c33-acc8-4f44-b4bd-fc10bd1cc35b |
+--------------------------------------+---------------------+--------------------------------------+
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ openstack network show d7b2a035-78d3-4525-a2e9-0841bbb09086
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2017-11-21T17:54:35Z |
| description | |
| id | d7b2a035-78d3-4525-a2e9-0841bbb09086 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| mtu | 1446 |
| name | research-network1 |
| port_security_enabled | True |
| project_id | b510e54c6feb48588ff99e9eff18b5a6 |
| project_id | b510e54c6feb48588ff99e9eff18b5a6 |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 45 |
| qos_policy_id | None |
| revision_number | 5 |
| router:external | Internal |
| shared | False |
| status | ACTIVE |
| subnets | 75f97c33-acc8-4f44-b4bd-fc10bd1cc35b |
| tags | [] |
| updated_at | 2017-11-21T17:55:57Z |
+---------------------------+--------------------------------------+
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$ openstack network show 33efb2c7-8c95-4d11-8662-405511021490
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2017-11-21T20:52:19Z |
| description | |
| id | 33efb2c7-8c95-4d11-8662-405511021490 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| mtu | 1496 |
| name | provider-172.25.250 |
| port_security_enabled | True |
| project_id | b510e54c6feb48588ff99e9eff18b5a6 |
| project_id | b510e54c6feb48588ff99e9eff18b5a6 |
| provider:network_type | vlan |
| provider:physical_network | datacentre |
| provider:segmentation_id | 500 |
| qos_policy_id | None |
| revision_number | 7 |
| router:external | External |
| shared | False |
| status | ACTIVE |
| subnets | b2442408-7390-4e08-906e-e99654318034 |
| tags | [] |
| updated_at | 2017-11-21T21:02:06Z |
+---------------------------+--------------------------------------+
[heat-admin@overcloud-controller-0 ~]$
[heat-admin@overcloud-controller-0 ~]$
虚拟机所在计算节点：
获取research-web1虚拟机ID
[root@overcloud-compute-0 heat-admin]# openstack server list --all-projects
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
| ID | Name | Status | Networks | Image Name |
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
| 637228e8-2d9c-43d8-9f1f-354930139745 | research-web1 | ACTIVE | research-network1=192.168.1.9, 172.25.250.109 | rhel7 |
| a1a3f218-1e85-47fc-b587-ec972695524e | finance-web1 | ACTIVE | finance-network1=192.168.1.10 | rhel7 |
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
[root@overcloud-compute-0 heat-admin]#
查询虚拟机所在主机
[root@overcloud-compute-0 heat-admin]# openstack server show 637228e8-2d9c-43d8-9f1f-354930139745 | grep host
| OS-EXT-SRV-ATTR:host | overcloud-compute-0.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-compute-0.localdomain |
| hostId | f9ae4023d0e55533979150fc7c28fc223771208564804b890d3c3016 |
[root@overcloud-compute-0 heat-admin]#
登陆虚拟机所在主机
[stack@director ~]$ openstack server list
+--------------------------------------+-------------------------+--------+------------------------+----------------+
| ID | Name | Status | Networks | Image Name |
+--------------------------------------+-------------------------+--------+------------------------+----------------+
| c3cc04ff-3a5e-47e9-afad-09e417ab47c4 | overcloud-compute-0 | ACTIVE | ctlplane=172.25.249.53 | overcloud-full |
| 2799c626-db04-4d63-b875-a96006a02de9 | overcloud-cephstorage-0 | ACTIVE | ctlplane=172.25.249.58 | overcloud-full |
| 9d03a91b-96cc-441e-af96-6e7343e6db92 | overcloud-controller-0 | ACTIVE | ctlplane=172.25.249.52 | overcloud-full |
+--------------------------------------+-------------------------+--------+------------------------+----------------+
[stack@director ~]$ ssh heat-admin@172.25.249.53
Last login: Tue Nov 21 23:33:05 2017 from 172.25.249.200
[heat-admin@overcloud-compute-0 ~]$
[heat-admin@overcloud-compute-0 ~]$
查询节点上的虚拟机
[root@overcloud-compute-0 heat-admin]# virsh list
Id Name State
----------------------------------------------------
2 instance-00000002 running
3 instance-00000003 running
[root@overcloud-compute-0 heat-admin]#
查出research-web1虚拟机（ID： 637228e8-2d9c-43d8-9f1f-354930139745）对应的名称：
[root@overcloud-compute-0 heat-admin]# virsh edit 3
<domain type='kvm'>
<name>instance-00000003</name>
<uuid>637228e8-2d9c-43d8-9f1f-354930139745</uuid>
<metadata>
//输入ESC,：q退出。
查询虚拟机使用的tap设备ID：(virsh edit/dumpxml 虚拟机No.)
virsh edit 3
输入/tap查询虚拟机tap设备与对应的linux bridge
<interface type='bridge'>
<mac address='fa:16:3e:fe:39:14'/>
<source bridge='qbr85e1ebd0-c7'/>
<target dev='tap85e1ebd0-c7'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
可见tap设备为tap85e1ebd0-c7，linux网桥为qbr85e1ebd0-c7。
通过以下命令可以查询到虚拟机的端口ID为85e1ebd0-c747-46bb-913b-2154493b8b3b，tap设备与linux网桥取端口ID前11位字符。
[root@overcloud-compute-0 heat-admin]# openstack server list --all-projects
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
| ID | Name | Status | Networks | Image Name |
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
| 637228e8-2d9c-43d8-9f1f-354930139745 | research-web1 | ACTIVE | research-network1=192.168.1.9, 172.25.250.109 | rhel7 |
| a1a3f218-1e85-47fc-b587-ec972695524e | finance-web1 | ACTIVE | finance-network1=192.168.1.10 | rhel7 |
+--------------------------------------+---------------+--------+-----------------------------------------------+------------+
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]# neutron port-list --device_id 637228e8-2d9c-43d8-9f1f-354930139745
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| 85e1ebd0-c747-46bb-913b-2154493b8b3b | | fa:16:3e:fe:39:14 | {"subnet_id": "75f97c33-acc8-4f44-b4bd-fc10bd1cc35b", "ip_address": "192.168.1.9"} |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]# nova interface-list 637228e8-2d9c-43d8-9f1f-354930139745
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| ACTIVE | 85e1ebd0-c747-46bb-913b-2154493b8b3b | d7b2a035-78d3-4525-a2e9-0841bbb09086 | 192.168.1.9 | fa:16:3e:fe:39:14 |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]# openstack port list | grep 192.168.1.9
| 85e1ebd0-c747-46bb-913b-2154493b8b3b | | fa:16:3e:fe:39:14 | ip_address='192.168.1.9', subnet_id='75f97c33-acc8-4f44-b4bd-fc10bd1cc35b' |
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]# openstack port list | grep 172.25.250.109
| 2864b06c-728b-47fb-aad2-07c2a80cd22b | | fa:16:3e:34:5b:09 | ip_address='172.25.250.109', subnet_id='b2442408-7390-4e08-906e-e99654318034' |
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]#
brctl show，查询对应的linux网桥，网桥名称qbr85e1ebd0-c7，接口名称qvb85e1ebd0-c7，虚拟机tap设备tap85e1ebd0-c7。
[root@overcloud-compute-0 heat-admin]# brctl show
bridge name bridge id STP enabled interfaces
qbr85e1ebd0-c7 8000.9e5ba70a29a4 no qvb85e1ebd0-c7
tap85e1ebd0-c7
qbrd0745089-3c 8000.cec797043f77 no qvbd0745089-3c
tapd0745089-3c
[root@overcloud-compute-0 heat-admin]#
ovs-vsctl show，查询ovs网桥，网桥br-int与接口名称qvo
网桥br-int，接口qvo85e1ebd0-c7，接口int-br-ex，接口patch-tun，内部接口br-int
网桥br-tun，接口patch-int，接口vxlan-ac180201，内部接口br-tun
网桥br-trunk，接口eth1，内部接口vlan10/20/30/br-trunk
网桥br-ex，接口phy-br-ex，内部接口br-ex
[root@overcloud-compute-0 heat-admin]# ovs-vsctl show
f90d01cc-1466-4968-acbe-8d45a9aa37c4
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-ac180201"
Interface "vxlan-ac180201"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.24.2.2", out_key=flow, remote_ip="172.24.2.1"}
Port br-tun
Interface br-tun
type: internal
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "qvo85e1ebd0-c7"
tag: 3
Interface "qvo85e1ebd0-c7"
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvod0745089-3c"
tag: 2
Interface "qvod0745089-3c"
Bridge br-trunk
fail_mode: standalone
Port "vlan30"
tag: 30
Interface "vlan30"
type: internal
Port "eth1"
Interface "eth1"
Port "vlan10"
tag: 10
Interface "vlan10"
type: internal
Port "vlan20"
tag: 20
Interface "vlan20"
type: internal
Port br-trunk
Interface br-trunk
type: internal
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-ex
Interface br-ex
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
ovs_version: "2.5.0"
[root@overcloud-compute-0 heat-admin]#
查看流表
[root@overcloud-compute-0 heat-admin]# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:000066766802b74f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-int): addr:16:5d:b7:15:de:e1
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(vxlan-ac180201): addr:96:11:1a:b1:63:88
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-tun): addr:66:76:68:02:b7:4f
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]#
[root@overcloud-compute-0 heat-admin]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x9557367520c86495, duration=80956.303s, table=0, n_packets=4553, n_bytes=429210, idle_age=120, hard_age=65534, priority=1,in_port=1 actions=resubmit(,2)
cookie=0x9557367520c86495, duration=80841.611s, table=0, n_packets=4325, n_bytes=18296294, idle_age=117, hard_age=65534, priority=1,in_port=2 actions=resubmit(,4)
cookie=0x9557367520c86495, duration=80956.303s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x9557367520c86495, duration=80956.302s, table=2, n_packets=4178, n_bytes=410304, idle_age=120, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
cookie=0x9557367520c86495, duration=80956.301s, table=2, n_packets=375, n_bytes=18906, idle_age=2881, hard_age=65534, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
cookie=0x9557367520c86495, duration=80956.300s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x9557367520c86495, duration=75113.931s, table=4, n_packets=2863, n_bytes=18130554, idle_age=6148, hard_age=65534, priority=1,tun_id=0x4e actions=mod_vlan_vid:2,resubmit(,10)
cookie=0x9557367520c86495, duration=15327.641s, table=4, n_packets=530, n_bytes=58415, idle_age=117, priority=1,tun_id=0x2d actions=mod_vlan_vid:3,resubmit(,10)
cookie=0x9557367520c86495, duration=80956.300s, table=4, n_packets=19, n_bytes=1582, idle_age=15671, hard_age=65534, priority=0 actions=drop
cookie=0x9557367520c86495, duration=80956.299s, table=6, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x9557367520c86495, duration=80956.299s, table=10, n_packets=4306, n_bytes=18294712, idle_age=117, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x9557367520c86495,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1
cookie=0x9557367520c86495, duration=192.853s, table=20, n_packets=34, n_bytes=2778, hard_timeout=300, idle_age=120, hard_age=117, priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:93:55:88 actions=load:0->NXM_OF_VLAN_TCI[],load:0x2d->NXM_NX_TUN_ID[],output:2
cookie=0x9557367520c86495, duration=80956.298s, table=20, n_packets=53, n_bytes=4339, idle_age=195, hard_age=65534, priority=0 actions=resubmit(,22)
cookie=0x9557367520c86495, duration=75113.932s, table=22, n_packets=380, n_bytes=18406, idle_age=2881, hard_age=65534, priority=1,dl_vlan=2 actions=strip_vlan,load:0x4e->NXM_NX_TUN_ID[],output:2
cookie=0x9557367520c86495, duration=15327.642s, table=22, n_packets=23, n_bytes=2298, idle_age=195, priority=1,dl_vlan=3 actions=strip_vlan,load:0x2d->NXM_NX_TUN_ID[],output:2
cookie=0x9557367520c86495, duration=80956.298s, table=22, n_packets=11, n_bytes=954, idle_age=15327, hard_age=65534, priority=0 actions=drop
[root@overcloud-compute-0 heat-admin]#
控制节点：
ovs-vsctl show，查看网桥与接口
网桥br-int，内部接口tap95567a30-aa，内部接口qg-79b6bf7d-6e，内部接口tap011bf55d-fd，内部接口tapae64b4aa-3f，内部接口qr-114a34e4-5e,接口int-br-ex，接口patch-tun，内部接口br-int
网桥br-tun，接口patch-int，接口vxlan-ac180202，内部接口br-tun
网桥br-trunk，接口eth1，内部接口vlan10/20/30/40/br-trunk
网桥br-ex，接口phy-br-ex，接口eth2，内部接口br-ex
[heat-admin@overcloud-controller-0 ~]$ ovs-vsctl show
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied)
[heat-admin@overcloud-controller-0 ~]$ sudo -i
[root@overcloud-controller-0 ~]#
[root@overcloud-controller-0 ~]# ovs-vsctl show
19c5af73-8404-4405-8571-713614ff3d46
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-ac180202"
Interface "vxlan-ac180202"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="172.24.2.1", out_key=flow, remote_ip="172.24.2.2"}
Port br-tun
Interface br-tun
type: internal
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port "tap95567a30-aa"
tag: 3
Interface "tap95567a30-aa"
type: internal
Port "qg-79b6bf7d-6e"
tag: 5
Interface "qg-79b6bf7d-6e"
type: internal
Port "tap011bf55d-fd"
tag: 1
Interface "tap011bf55d-fd"
type: internal
Port "tapae64b4aa-3f"
tag: 4
Interface "tapae64b4aa-3f"
type: internal
Port "qr-114a34e4-5e"
tag: 4
Interface "qr-114a34e4-5e"
type: internal
Bridge br-trunk
fail_mode: standalone
Port br-trunk
Interface br-trunk
type: internal
Port "vlan20"
tag: 20
Interface "vlan20"
type: internal
Port "vlan30"
tag: 30
Interface "vlan30"
type: internal
Port "vlan10"
tag: 10
Interface "vlan10"
type: internal
Port "eth1"
Interface "eth1"
Port "vlan40"
tag: 40
Interface "vlan40"
type: internal
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-ex
Interface br-ex
type: internal
Port "eth2"
Interface "eth2"
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
ovs_version: "2.5.0"
[root@overcloud-controller-0 ~]#
ip netns list，查看命名空间
[root@overcloud-controller-0 ~]# ip netns list
qrouter-f5c7f658-d41a-4340-b5c3-c99ecc8d1c5e
qdhcp-d7b2a035-78d3-4525-a2e9-0841bbb09086
qdhcp-2ad72164-1f03-4678-a953-855da36750f4
qdhcp-02471475-7999-4bf7-8b9d-2426e0cd83da
[root@overcloud-controller-0 ~]#
ip netns exec 命名空间名称 /bin/bash，进入两个命名空间查看。
然后执行ip address show，可以得到命名空间的接口名称，结合ovs-vsctl show可以知道router接口通过的vlan分别是多少。floating IP，从控制节点通过浮动IP访问计算节点上的ip。
exit退出。
[root@overcloud-controller-0 ~]# ip netns exec qrouter-f5c7f658-d41a-4340-b5c3-c99ecc8d1c5e /bin/bash
[root@overcloud-controller-0 ~]# ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
26: qr-114a34e4-5e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:93:55:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global qr-114a34e4-5e
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe93:5588/64 scope link
valid_lft forever preferred_lft forever
27: qg-79b6bf7d-6e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:cd:7a:a5 brd ff:ff:ff:ff:ff:ff
inet 172.25.250.103/24 brd 172.25.250.255 scope global qg-79b6bf7d-6e
valid_lft forever preferred_lft forever
inet 172.25.250.109/32 brd 172.25.250.109 scope global qg-79b6bf7d-6e
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fecd:7aa5/64 scope link
valid_lft forever preferred_lft forever
[root@overcloud-controller-0 ~]# exit
exit
[root@overcloud-controller-0 ~]#
172.25.250.103是router的ip。
[root@overcloud-controller-0 heat-admin]# source overcloudrc
[root@overcloud-controller-0 heat-admin]#
[root@overcloud-controller-0 heat-admin]# neutron port-list | grep 172.25.250.103
| 79b6bf7d-6eda-43a7-b4d1-41a9e688d1f4 | | fa:16:3e:cd:7a:a5 | {"subnet_id": "b2442408-7390-4e08-906e-e99654318034", "ip_address": "172.25.250.103"} |
[root@overcloud-controller-0 heat-admin]# neutron port-show 79b6bf7d-6eda-43a7-b4d1-41a9e688d1f4
+-----------------------+---------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+---------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | overcloud-controller-0.localdomain |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2017-11-21T21:11:02Z |
| description | |
| device_id | f5c7f658-d41a-4340-b5c3-c99ecc8d1c5e |
| device_owner | network:router_gateway |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "b2442408-7390-4e08-906e-e99654318034", "ip_address": "172.25.250.103"} |
| id | 79b6bf7d-6eda-43a7-b4d1-41a9e688d1f4 |
| mac_address | fa:16:3e:cd:7a:a5 |
| name | |
| network_id | 33efb2c7-8c95-4d11-8662-405511021490 |
| port_security_enabled | False |
| project_id | |
| qos_policy_id | |
| revision_number | 7 |
| security_groups | |
| status | ACTIVE |
| tenant_id | |
| updated_at | 2017-11-21T21:11:04Z |
+-----------------------+---------------------------------------------------------------------------------------+
[root@overcloud-controller-0 heat-admin]#
查看dhcp ns：
[root@overcloud-controller-0 heat-admin]# ip netns exec qdhcp-d7b2a035-78d3-4525-a2e9-0841bbb09086 /bin/bash
[root@overcloud-controller-0 heat-admin]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
24: tapae64b4aa-3f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:5a:8d:87 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.1.255 scope global tapae64b4aa-3f
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe5a:8d87/64 scope link
valid_lft forever preferred_lft forever
[root@overcloud-controller-0 heat-admin]# exit
exit
[root@overcloud-controller-0 heat-admin]# ip netns exec qdhcp-2ad72164-1f03-4678-a953-855da36750f4 /bin/bash
[root@overcloud-controller-0 heat-admin]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
21: tap95567a30-aa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:0a:4f:c1 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.1.255 scope global tap95567a30-aa
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe0a:4fc1/64 scope link
valid_lft forever preferred_lft forever
[root@overcloud-controller-0 heat-admin]#
[root@overcloud-controller-0 heat-admin]# exit
exit
[root@overcloud-controller-0 heat-admin]#
[root@overcloud-controller-0 heat-admin]# ip netns exec qdhcp-02471475-7999-4bf7-8b9d-2426e0cd83da /bin/bash
[root@overcloud-controller-0 heat-admin]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
20: tap011bf55d-fd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
link/ether fa:16:3e:a4:cb:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.1.255 scope global tap011bf55d-fd
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fea4:cb03/64 scope link
valid_lft forever preferred_lft forever
[root@overcloud-controller-0 heat-admin]# exit
exit
[root@overcloud-controller-0 heat-admin]#
查看流表
[root@overcloud-controller-0 ~]# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000ea45e2083b46
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-int): addr:46:73:5c:f6:bf:04
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(vxlan-ac180202): addr:2a:3e:9e:99:84:11
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-tun): addr:ea:45:e2:08:3b:46
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
[root@overcloud-controller-0 ~]#
[root@overcloud-controller-0 ~]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x8616e9b81f19fa23, duration=80939.517s, table=0, n_packets=5214, n_bytes=18341120, idle_age=121, hard_age=65534, priority=1,in_port=1 actions=resubmit(,2)
cookie=0x8616e9b81f19fa23, duration=80845.334s, table=0, n_packets=4542, n_bytes=428256, idle_age=124, hard_age=65534, priority=1,in_port=2 actions=resubmit(,4)
cookie=0x8616e9b81f19fa23, duration=80939.516s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x8616e9b81f19fa23, duration=80939.514s, table=2, n_packets=4297, n_bytes=18296278, idle_age=121, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
cookie=0x8616e9b81f19fa23, duration=80939.513s, table=2, n_packets=917, n_bytes=44842, idle_age=122, hard_age=65534, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
cookie=0x8616e9b81f19fa23, duration=80939.512s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x8616e9b81f19fa23, duration=80846.085s, table=4, n_packets=3041, n_bytes=278016, idle_age=2885, hard_age=65534, priority=1,tun_id=0x4e actions=mod_vlan_vid:1,resubmit(,10)
cookie=0x8616e9b81f19fa23, duration=80454.899s, table=4, n_packets=911, n_bytes=99243, idle_age=65534, hard_age=65534, priority=1,tun_id=0x5a actions=mod_vlan_vid:3,resubmit(,10)
cookie=0x8616e9b81f19fa23, duration=27302.104s, table=4, n_packets=590, n_bytes=50997, idle_age=124, priority=1,tun_id=0x2d actions=mod_vlan_vid:4,resubmit(,10)
cookie=0x8616e9b81f19fa23, duration=80939.511s, table=4, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x8616e9b81f19fa23, duration=80939.511s, table=6, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x8616e9b81f19fa23, duration=80939.510s, table=10, n_packets=4542, n_bytes=428256, idle_age=124, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x8616e9b81f19fa23,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:1
cookie=0x8616e9b81f19fa23, duration=199.583s, table=20, n_packets=36, n_bytes=3832, hard_timeout=300, idle_age=121, hard_age=124, priority=1,vlan_tci=0x0004/0x0fff,dl_dst=fa:16:3e:fe:39:14 actions=load:0->NXM_OF_VLAN_TCI[],load:0x2d->NXM_NX_TUN_ID[],output:2
cookie=0x8616e9b81f19fa23, duration=80939.509s, table=20, n_packets=123, n_bytes=10278, idle_age=27467, hard_age=65534, priority=0 actions=resubmit(,22)
cookie=0x8616e9b81f19fa23, duration=80845.332s, table=22, n_packets=4, n_bytes=320, idle_age=65534, hard_age=65534, priority=1,dl_vlan=1 actions=strip_vlan,load:0x4e->NXM_NX_TUN_ID[],output:2
cookie=0x8616e9b81f19fa23, duration=80454.900s, table=22, n_packets=6, n_bytes=468, idle_age=65534, hard_age=65534, priority=1,dl_vlan=3 actions=strip_vlan,load:0x5a->NXM_NX_TUN_ID[],output:2
cookie=0x8616e9b81f19fa23, duration=27302.105s, table=22, n_packets=11, n_bytes=974, idle_age=15675, priority=1,dl_vlan=4 actions=strip_vlan,load:0x2d->NXM_NX_TUN_ID[],output:2
cookie=0x8616e9b81f19fa23, duration=80939.508s, table=22, n_packets=1019, n_bytes=53358, idle_age=122, hard_age=65534, priority=0 actions=drop
[root@overcloud-controller-0 ~]#
通过以上分析，输出下图
实验环境网桥信息.jpg
L2与L3网络
计算节点上两个虚拟机2层互通，通过各自计算节点的里neutron-openvswitch-agent互通。
l2网络互通.png
计算节点上两个虚拟机3层互通，通过各自计算节点的neutron-openvswitch-agent连接到网络节点上的neutron-l3-agent后互通。
l3网络互通.png
抓包与查看ip table。
tcpdump -i tab
iptable命令
<未完>
