warden建立容器的過程

 

基礎知識（本節轉自http://www.54chen.com/architecture/cloud-foundry-warden-part1.html）

rv = unshare(CLONE_NEWNS);

     unshare這個調用，可以把掛載的檔案系統設定成只在新的掛載命名空間（mount namespace）中可見。

execvp(argv[0], argv);

      execvp()會從PATH 環境變數所指的目錄中尋找符合參數file 的檔案名稱，找到後便執行該檔案，然後將第二個參數argv傳給該欲執行的檔案。

shopt -s nullglob

      設定shell環境變數nullglob的值為on，nullglob為on時對於萬用字元匹配時，若匹配不到時為空白（相對應的為萬用字元本身）。

int stat(const char *restrict pathname, struct stat *restrict buf);

      提供檔案名稱字，擷取檔案對應屬性。

build-essential軟體包

      作用是提供編譯器必須軟體包的列表資訊，也就是說編譯器有了這個軟體包，它才知道 標頭檔在哪，才知道庫函數在哪，還會下載依賴的軟體包，最後才組成一個開發環境。

工具debootstrap

       可以用於在系統的某個目錄中安裝一套基本系統，這個基本系統除了一些配置項外，與ubuntu安裝程式在安裝的第一階段安裝的內容基本相同。這項功能有許多有趣的功能，例如，你可以從某個定製版本的ubuntu Live光碟片上通過這個命令快速的在硬碟上安裝ubuntu而不需要ubuntu的安裝程式，也可以把建立在硬碟上的基本系統目錄作為某些涉及系統安全性服務的chroot運行環境，通過chroot進入該目錄並調試和運行一些可能修改系統配置的應用程式，作為定製小型系統模板等等。

aufs

       一種檔案格式，可以mount到目錄，同時控制唯讀和讀寫。

overlayfs

         另一種檔案格式，在ubuntu 11.04後開始替代aufs作為官方livecd的檔案格式。  cgroup的初始化初始化的指令碼為：

cgroup_path=/sys/fs/cgroupmount -t tmpfs none $cgroup_pathfor subsystem in cpu cpuacct devices memorydo  mkdir -p $cgroup_path/$subsystem   if ! grep -q "${cgroup_path}/$subsystem " /proc/mounts  then    mount -t cgroup -o $subsystem none $cgroup_path/$subsystem  fidone

 上面的命令執行完後，可以看下掛載的結果： 

# grep '^cgroup' /proc/mountscgroups /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0

 網路的控制

 Every container is assigned a network interface which is one side of a
 virtual ethernet pair created on the host. The other side of the virtual
 ethernet pair is only visible on the host (from the root namespace).
 The pair is configured to use IPs in a small and static subnet. Traffic
 from and to the container can be forwarded using NAT. Additionally, all
 traffic can be filtered and shaped as needed, using readily available
 tools such as `iptables`.網路初始化

 

echo 1 > /proc/sys/net/ipv4/ip_forward # iptables-save | grep -E 'warden|\*'
*nat
:warden-instance-16al6hojp15 - [0:0]
:warden-prerouting - [0:0]
-A PREROUTING -i eth0 -j warden-prerouting
-A OUTPUT -o lo -j warden-prerouting
-A warden-prerouting -j warden-instance-16al6hojp15
*mangle
*filter
:warden-default - [0:0]
:warden-dispatch - [0:0]
:warden-instance-16al6hojp15 - [0:0]
-A INPUT -i w-+ -j warden-dispatch
-A FORWARD -i w-+ -j warden-dispatch
-A warden-dispatch -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A warden-dispatch -i w-16al6hojp15-0 -g warden-instance-16al6hojp15
-A warden-dispatch -j DROP
-A warden-instance-16al6hojp15 -g warden-default

  停止apparmor：      /etc/init.d/apparmor teardown
 

# quotaon(8) exits with non-zero status when quotas are ENABLEDif quotaon -p $CONTAINER_DEPOT_MOUNT_POINT_PATH > /dev/nullthen  mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH  quotacheck -ugmb -F vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH  quotaon $CONTAINER_DEPOT_MOUNT_POINT_PATHfi

  

 檔案系統Every container gets a private root filesystem. This filesystem iscreated by stacking a read-only filesytem and a read-write filesystem.
This is implemented by using `aufs` on Ubuntu versions from 10.04 up to
11.10, and `overlayfs` on Ubuntu 12.04.

The read-only filesystem contains the minimal set of Ubuntu packages and
Warden-specific modifications common to all containers. The read-write
filesystem stores files overriding container-specific settings when
necessary. Because all writes are applied to the read-write filesystem,
containers can share the same read-only base filesystem.

The read-write filesystem is created by formatting a large sparse file.
Because the size of this file is fixed, the filesystem that it contains
cannot grow beyond this initial size.

  建立一個容器set -o nounset
set -o errexit
shopt -s nullglob cp -r skeleton "${target}"
unshare -m "${target}"/setup.sh "${target}"/setup.sh指令碼 1. 首先產生一個設定檔 id=16al6hojp15
network_netmask=255.255.255.252
network_host_ip=10.254.0.17
network_host_iface=w-16al6hojp15-0
network_container_ip=10.254.0.18
network_container_iface=w-16al6hojp15-1
user_uid=10004 2. 調用setup_fs() 

mkdir -p rootfs ${target}

mount -n -t overlayfs -o rw,upperdir=rootfs,lowerdir=${1} none ${target}

3. 調用prepare.sh 刪除不需要的檔案，並建立必要的裝置

 4. 配置系統參數 # cat etc/hosts
127.0.0.1 16al6hojp15 localhost
10.254.0.17 host
10.254.0.18 container # cat etc/hostname
16al6hojp15 # cat etc/network/interfaces
auto lo
iface lo inet loopback
auto w-16al6hojp15-1
iface w-16al6hojp15-1 inet static
  gateway 10.254.0.17
  address 10.254.0.18
  netmask 255.255.255.252 cp /etc/resolv.conf ${target}/etc/ chroot並添加使用者：useradd -mU -u ${user_uid} -s /bin/bash vcap # Copy override directory     其實就是幾個etc檔案和一個sbin/warden-stop.sh檔案
cp -r override/* ${target}/
chmod 700 ${target}/sbin/warden-* 5. 配置ssh 6. 配置mesg # The `mesg` tool modifies permissions on stdin. Warden regularly passes a
# custom stdin, which makes `mesg` complain that stdin is not a tty. Instead of
# removing all occurances of `mesg`, we simply bind it to /bin/true.
chroot <<EOS
rm /usr/bin/mesg
ln -s /bin/true /usr/bin/mesg
EOS 7. 物理機上添加網卡作為容器的網關：      ifconfig ${network_host_iface} ${network_host_ip} netmask ${network_netmask} 並添加tc規則控制頻寬：      qdisc tbf 8002: dev w-16al6hojp15-0 root refcnt 2 rate 8192bit burst 9b lat 24.4ms 

     qdisc ingress ffff: dev w-16al6hojp15-0 parent ffff:fff1 ---------------- 

 8. 配置cgroup 

# Add new group for every subsystemfor system_path in /sys/fs/cgroup/*do  instance_path=$system_path/instance-$id   mkdir -p $instance_path   if [ $(basename $system_path) == "cpuset" ]  then    cat $system_path/cpuset.cpus > $instance_path/cpuset.cpus    cat $system_path/cpuset.mems > $instance_path/cpuset.mems  fi   echo 1 > $instance_path/cgroup.clone_children  echo $PID > $instance_path/tasksdone echo ${PPID} >> ppid ip link add name ${network_host_iface} type veth peer name ${network_container_iface}ip link set ${network_host_iface} netns 1ip link set ${network_container_iface} netns ${PID}

 建立好後，可以通過ssh -i access_key root@10.254.0.18 來登陸，密碼是前面安裝時設定的密碼