OpenStack Development Step-by-Step

Set up your Open Stack

There was several ways to deploy OpenStack, Devstack was easily for developer to deploy Open Stack.

Please follow the this URL to the set up your Open Stack.

http://docs.openstack.org/developer/devstack/

nested KVM support:

• Kernel
  • First check if your system has already configured nested KVM:

[Email protected]:~# dpkg-s/etc/modprobe.d/qemu-system-x86.confqemu-system-x86:/etc/modprobe.d/ Qemu-system-x86.conf[email protected]:~# cat/etc/modprobe.d/qemu-system-x86.confoptions Kvm_intel nested=1

• If not, run:

echo "Options Kvm-intel nested=1″| sudo tee/etc/modprobe.d/kvm-intel.conf

• Then reload Kvm_intel.ko

Rmmod Kvm_intelmodprobe Kvm_intel

• Nova
  • Optional:modify nova.conf and add Host-passthrough. Please note that it may impact migration.

Libvirt_cpu_mode = Host-passthrough

• Restart Nova Services.

PCI PassThrough Support

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/ Chap-virtualization-pci_passthrough.html

There is ways to the support PCI PassThrough.

1. Legacy UIO Framework (PCI Assign);
2. Vfio, it is recommended http://lwn.net/Articles/509153/

For PCI SR-Iov Device we have a specially.

Preparation for PCI PassThrough

(1) Enable the Intel vt-d extensions Http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Setup in your BIOS. Check you have enable it.

$sudo DMESG | GREP-E DMAR-E IOMMU

(2) Activate Intel vt-d in the kernel http://www.zerg.sc/technote/?p=241

$ sudo echo 1 >/sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Update the kernel cmdline.

Https://www.gnu.org/software/grub/manual/html_node/Simple-configuration.html

$ sudo su$ sudo sed-i-e ' s/\ (^grub_cmdline_linux_default.*\) "$/\1 intel_iommu=on"/'/etc/default/grub$ sudo update-grub $ sudo grep intel_iommu=on/boot/grub/grub.cfg

Use Legacy UIO framework instead of Vfio

Why we still need the legacy UIO framework, for there is some thing wrong with Vfio and IGB in some PC.

Http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Set parameter KVM allow_unsafe_assigned_interrupts

Transient Configure (take effect Immediately)

$ sudo echo 1 >/sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Or Persistent

$ sudo echo "options KVM Allow_unsafe_assigned_interrupts=1" >>  /etc/modprobe.d/kvm_iommu.conf

Search the PCI device

$ lspci-n01:00.0 0200:8086:10FB (rev.) 01:00.1 0200:8086:10FB (rev.) $ lspci01:00.0 Ethernet Controller:intel Corpor ation 82599ES 10-gigabit sfi/sfp+ Network Connection (rev.) 01:00.1 Ethernet Controller:intel Corporation 82599ES 10-gig Abit sfi/sfp+ Network Connection (rev 01)

Bind pci_stub driver to PCI device

$ sudo modprobe pci_stub$ sudo su (take effect Immediately) $ echo "8086 10FB" >/sys/bus/pci/drivers/pci-stub/new_id$ EC Ho 0000:01:00.0   >/sys/bus/pci/devices/0000\:01\:00.0/driver/unbind$ echo 0000:01:00.0  >/sys/bus/pci /drivers/pci-stub/bind$ echo 0000:01:00.1   >/sys/bus/pci/devices/0000\:01\:00.1/driver/unbind$ echo 0000:01:00.1  >/sys/bus/pci/drivers/pci-stub/bind

Start attach PCI device to guest and start

1. In this-the-to-use Virsh-to-Start your guest, you must assign the driver name, or you'll failed fo want R Libvirt would use Vfio driver as default.
2. Name= ' KVM ' means:the legacy PCI device assignment handled directly by the KVM kernel module

$ echo "
Use  vfio 
There is tons of tutorials on what to get vt-d working with KVM/LIBVIRT/QEMU/ETC. It's still a fairly new and unused feature, which means that it's getting updated pretty frequently. Unfortunately, the Internet doesn ' t update as quickly, so that most of the documentation out there are not accurate or the best Approach. Readlink You can refer if any trouble.
http://hinish.com/?p=338
We need this command to load the kernel modules at boot time
$ sudo echo "Vfiovfio_iommu_type1vfio_pci" >>/etc/modules
Set vfio_iommu_type1 allow_unsafe_interrupts Parameters If you ' re overriding the IVRs table to enable intremap you dont NE Ed this line persistent (need reboot.)
$ echo "Options vfio_iommu_type1 Allow_unsafe_interrupts=1" >/etc/modprobe.d/vfio_iommu_type1.conf
Or transient (take effect Immediately)
$ sudo echo 1 >  /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
Disable AppArmor for LIBVIRTD
Actually, we don't need to disable Apparmor, the This is a bug on Ubuntu, we need to work around it. (It can not works)
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1276719
Http://libvirt.org/drvqemu.html#securitysvirtaa
$ apt-get Install apparmor-utils$ sudo aa-complain/usr/sbin/libvirtd$ sudo su$ for x in/etc/apparmor.d/libvirt/libvirt-* [0-9a-b];d o aa-complain $x; Done
Or (It can not works)
$ sudo ln-s  /etc/apparmor.d/usr.sbin.libvirtd/etc/apparmor.d/disable/$ sudo ln-s/etc/apparmor.d/ usr.lib.libvirt.virt-aa-helper/etc/apparmor.d/disable/$ sudo apparmor_parser-r/etc/apparmor.d/usr.sbin.libvirtd$ sudo apparmor_parser-r/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper$ sudo service libvirt-bin restart
Or Remove it (the best)
http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/
$ sudo/etc/init.d/apparmor stop$ sudo update-rc.d-f apparmor remove$ sudo apt-get--purge remove AppArmor apparmor-utils Libapparmor-perl Libapparmor1
It may cause the Ubuntu-desktop crash, please reinstall it.
$ sudo apt-get install Ubuntu-desktop
Find the all the PCI devices in Sam Iommu Group
We must binding all devices on same group to the VFIO-PCI driver creates.
$ readlink/sys/bus/pci/devices/0000\:01\:00.0/iommu_group.. /.. /.. /.. /kernel/iommu_groups/1$ ls-l/sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices
Bind Vfio driver to PCI device
$ sudo su$ echo "8086 10fb" >/sys/bus/pci/drivers/vfio-pci/new_id$ echo "0000:01:00.1" >/sys/bus/pci/devices/0000 \:01\:00.1/driver/unbind$ echo "0000:01:00.1" >/sys/bus/pci/drivers/vfio-pci/bind$ echo "0000:01:00.0" >/sys/ bus/pci/devices/0000\:01\:00.0/driver/unbind$ echo "0000:01:00.0" >/sys/bus/pci/drivers/vfio-pci/bind
Start attach PCI device to guest and start
$ echo "
Configure for IGB module for SR-Iov support.
Https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_ Configuration_and_guest_installation_guide/sect-virtualization_host_configuration_and_guest_installation_ Guide-sr_iov-how_sr_iov_libvirt_works.html
Persistent configure for IGB module.
https://communities.intel.com/community/wired/blog/2011/08/31/ Using-sr-iov-with-intel-ethernet-and-red-hat-enterprise-linux
$ sudo echo ' options igb max_vfs=7 ' >/etc/modprobe.d/igb.conf$ sudo echo ' igb ' >>/etc/modules$ sudo update-init Ramfs-k all-t-U
Transient Configure for IGB module.
$ sudo su$ sudo rmmod igb && sudo modprobe igb max_vfs=7
It Maill failed as follow on PC:
Jan 04:42:18 localhost kernel: [8441.914658] Ixgbe 0000:01:00.1:sr-iov:bus number out of Rangejan 04:42:18 Localh OST kernel: [8441.914662] Ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI Sriov: 12
This is Intel official document and it is some different from the above.
http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ Ethernet-x520-sr-iov-red-hat-tech-brief.html
Work around if Pci-passthrough can isn't work.
 Upgrade your kernel. 
Download the latest stable kernel. https://www.kernel.org/
$ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.1.tar.xz$ tar-xvf linux-3.18.1.tar.xz$ CD linux-3.18.1$ cp/boot/config-3.13.0-43-generic/.config$ echo "" | Make Oldconfig
Please enable Config_ip_nf_nat the CONFIG, kernel 3.17 Change the macro define for NAT.
$ sed-i-E "s/# config_ip_nf_nat.*/config_ip_nf_nat=m/"./.config$ make-j8$ make modules_install$ make install$ sudo reb Oot
 Compile your Owen qemu 
$ git clone git://git.qemu.org/qemu.git$ apt-get install Libglib2.0-dev Zlib1g-dev Libspice-server-dev libspice-protocol-dev$./configure '--target-list=x86_64-softmmu '--enable-debug '--ENABLE-KVM '--enable-spice ' --prefix=/home/shhfeng/qemu/' $ make
 re  make your INITRD
$ gunzip # uncompress your initrd-*.img file $ mkdir tmp$ cd tmp/$ cpio-i--make-directories < "your uncompress in  ITRD direction. " $ tree-c.
# tree:option-c Color show,-n shutdown color show.
Or
$ cd tmp/$ GZIP-DC. /sth.gz |cpio-id
# Edit Your INITRD,
$ CD tmp/
# Add or Remove *.ko
$ Find./| Cpio-h newc-o >/boot/initrd-2.6.22.6.img-ahci$ gzip/boot/initrd-2.6.22.6.img-ahci$ MV Initrd-2.6.22.6.img-ahci.gz/boot/initrd-2.6.22.6.img-ahci
 Build up your skills to develop Open stack 
 Doc in Repository 
Please read the Docs in/opt/stack/nova/doc/source/devref first. Then you'll get some ideas about Open Stack. Read up these files, you can ignore them if you already know them.
Http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start

 
  

  
   
 
    
Gerrit.rst
 
   
 
  
It introduce the Gerrit work flow.

 
  

  
   
 
    
Development.environment.rst
 
   
 
  
It introduce how to set up development environment.

 
  

  
   
 
    
Architecture.rst
 
   
 
  
Introduce the profile of the Open Stack.

 
  

  
   
 
    
Rpc.rst
 
   
 
  
Open Stack RPC is different with the traditional SUN IPC, it base on a AMQP broker. Http://www.rabbitmq.com/getstarted.html
Learn more about implementation of RPC. Please seen the Open Stack Common Library.
$ git clone git://git.openstack.org/openstack/oslo.git
Also Learn more about Kombu:
http://kombu.readthedocs.org/en/latest/

 
  

  
   
 
    
Threading.rst
 
   
 
  
All OpenStack services use *green thread* model of threading, If your know coroutine and you'll learn greenthread.
All the above files also can is access by this URL link.
Http://docs.openstack.org/developer/nova/devref/rpc.html
Import Component
You had better does some exercise about some technologies that Open Stack uses. This is the helpful for your to know Open Stack.

 
  

  
   
 
    
RABBITMQ Python Tutorials
 
   
 
  
Tutorial:
Https://github.com/rabbitmq/rabbitmq-tutorials/http://wenku.baidu.com/view/800285ea0975f46527d3e19b.html (A Chinese document for RABBITMQ) http://www.openstack.cn/p136.html (a Chinese simple Document for RABBITMQ)
Tutorial Source Code:
https://github.com/rabbitmq/rabbitmq-tutorials/
 
  

  
   
 
    
SQLAlchemy (http://www.sqlalchemy.org/)
 
   
 
  
Tutorial Source Code:
Http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html
 
  

  
   
 
    
WSGI and Eventlet
 
   
 
  
Read the Red hat REST API Guide. It is the useful to understand REST API. Https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.0/html-single/REST_API_Guide /index.html#sect-rest_api_guide-common_features-resources-creating_resources
Join the Open Stack Community to do contributions
 
  

  
   
 
    
Join the IRC
 
   
 
  
Irc:https://wiki.openstack.org/wiki/irc

 
  

  
   
 
    
Subscribe the Mail-list
 
   
 
  
Dev:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
General:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

 
  

  
   
 
    
To submit the patch
 
   
 
  
Workflow:https://wiki.openstack.org/wiki/gerrit_workflow
Play with Open Stack
DashBoard
Http://docs.openstack.org/user-guide/content/log_in_dashboard.html
Try to access http://localhost/
 CURL 
Http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start
I have the write a small script to get token ID, as follow, you can name it as "Get-token":
#!/bin/bashreturn= ' curl-i ' http://127.0.0.1:5000/v2.0/tokens '-x post-h "Content-type:application/json"-H "Accept: Application/json "-  d ' {" auth ": {" Tenantname ":" admin "," passwordcredentials ": {" username ":" admin "," Password ":" 123 "}} '" Echo $returnecho "-----------------------------------------------------------------------------" id= "" for x in $return; Do    if ["x$id" = "Xtokenid"] and then         id= ' echo $x | tr-d ' [",] '        export token= $id        echo ' token= $id '        id= ' "        Break    fi    If [" x$x "= ' x ' id": '], then         id= "Tokenid"    Fidoneecho ""
This script just are a sample, you can improve it.
Run it by
$ . Get-token
Then you can run other commands with the TOKEN environment value, such as get all the tenents, as follow:
$ curl-i-X GET Http://127.0.0.1:35357/v2.0/tenants-H "user-agent:python-keystoneclient"-H "X-auth-token: $TOKEN"
 Open Stack Client 
Http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start
You should export the Os_username and Os_tenant_name environment by the follow command firstly.
$ cd devstack/$. OPENRC Os_username Os_tenant_name
Http://docs.openstack.org/juno/config-reference/content/section_compute-scheduler.html
Update your/etc/nova/nova.conf
ADD the follow option to ' [DEFAULT] ' section.
$ grep-e default-e pci_passthrough-e pci_alias-e scheduler_available_filters/etc/nova/nova.conf [DEFAULT]pci_passthr ough_whitelist={"product_id": "10FB", "vendor_id": "8086", "DeviceIDs": "0000:01:00.0, 0000:01:00.1"}pci_passthrough _whitelist={"product_id": "8c20", "vendor_id": "8086", "DeviceIDs": "0000:00:1b.0"}pci_alias = {"vendor_id": "8086", " product_id ":" 10FB "," name ":" intelnic "}scheduler_available_filters = Nova.scheduler.filters.pci_passthrough_filter . Pcipassthroughfilter
Create A instance with Pci-passthrough device.
Https://wiki.openstack.org/wiki/Pci_passthrough
$ Nova flavor-list$ Nova Flavor-key m1.tiny set pci_passthrough:alias=intelnic:1$ Nova boot  PCI  --flavor m1.tiny  --IMAGE=CIRROS-0.3.2-X86_64-UEC
Debug with Open Stack
I usually use the "print" for debugging. The "print" is on the stack, screen terminal.
$ cd devstack/$./rejoin-stack.sh
Or
$ screen–lsthere is a screen on:13846.stack$ screen-r stack
Some Useful operation:
Run a command by ' Ctrl + A +: ', you  can input help command to show more info. Swith to any window by "Ctrl + A + '  ", input the Windows number want to switch. Swith to all window by "Ctrl + A +"  and select the Windows want to switch.
Write TestCase
Https://wiki.openstack.org/wiki/SmallTestingGuide
OpenStack Development Step-by-Step