xm migrate源碼分析
xen動態遷移虛機的命令為：xm migrate --live <domain id> <destination machine>
遷移的原理
Xen live migration begins by sending a request, or reservation, to the target specifying the resources the migrating domain will need. If the target accepts the request, the source begins the iterative precopy phase of migration. During this step, Xen copies pages of memory over a TCP connection to the destination host. While this is happening, pages that change are marked as dirty and then recopied. The machine iterates this until only very frequently changed pages remain, at which point it begins the stop and copy phase. Now Xen stops the VM and copies over any pages that change too frequently to efficiently copy during the previous phase. In practice, our testing suggests that Xen usually reaches this point after four to eight iterations. Finally the VM starts executing on the new machine.
By default, Xen will iterate up to 29 times and stop if the number of dirty pages falls below a certain threshold. You can specify this threshold and the number of iterations at compile time, but the defaults should work fine.
xen動態遷移分為save和restore兩部分，先看save部分。tools/python/xen/xm/migrate.py
該函數主要是對輸入的命令作參數解析，然後跳轉到 server.xend.domain.migrate(dom, dst, opts.vals.live,
opts.vals.port,
opts.vals.node,
opts.vals.ssl) -->XendDomain.domain_migrate()tools/python/xen/xend/XendDomain.pydomain_migrate(self, domid, dst, live=False, port=0, node=-1, ssl=None):1. Make sure vm existing and being running dominfo = self.domain_lookup_nr(domid) # get a structure of XendDomainInfo
if not dominfo:
raise XendInvalidDomain(str(domid))
if dominfo.getDomid() == DOM0_ID:
raise XendError("Cannot migrate privileged domain %s" % domid)
if dominfo._stateGet() != DOM_STATE_RUNNING:
raise VMBadState("Domain is not running",
POWER_STATE_NAMES[DOM_STATE_RUNNING],POWER_STATE_NAMES[dominfo._stateGet()])2. Notify all device about intention of migration
dominfo.testMigrateDevices(True, dst) ---> XendDomainInfo.migrateDevice(n, c, network, dst, DEV_MIGRATE_TEST, self.getName()) ---> XendDomainInfo.getDeviceController(deviceClass).migrate(deviceConfig,
network, dst, step, domName)
---> DevController.migrate() 由於xoptions.get_external_migration_tool()返回為空白，實際上什麼也不做，直接返回0 This function is called for 4 steps: If step == 0: Check whether the device is ready to be migrated
or can at all be migrated; return a '-1' if
the device is NOT ready, a '0' otherwise. If it is
not ready ( = not possible to migrate this device),
migration will not take place.
step == 1: Called immediately after step 0; migration
of the kernel has started;
step == 2: Called after the suspend has been issued
to the domain and the domain is not scheduled anymore.
Synchronize with what was started in step 1, if necessary.
Now the device should initiate its transfer to the
given target. Since there might be more than just
one device initiating a migration, this step should
put the process performing the transfer into the
background and return immediately to achieve as much
concurrency as possible.
step == 3: Synchronize with the migration of the device that
was initiated in step 2.
Make sure that the migration has finished and only
then return from the call. 這兒DEV_MIGRATE_TEST=0，即步驟03. For live migration, make sure there's memory free for enabling shadow mode dominfo.checkLiveMigrateMemory() --->XendDomainInfo.checkLiveMigrateMemory() 遷移需要的記憶體為：1MB per vcpu plus 4Kib/Mib of RAM 這些記憶體通過balloon.free(overhead_kb, self)獲得4. 如果使用--ssl選項，則建立SSL串連；否則，建立普通tcp串連5. 開始遷移 XendCheckpoint.save(sock.fileno(), dominfo, True, live, dst, node=node) --->XendCheckpoint.save()tools/python/xen/xend/XendCheckPoint.py def save(fd, dominfo, network, live, dst, checkpoint=False, node=-1) 1. 首先發送SIGNATURE，即LinuxGuestRecord 2. 為了避免將虛機遷移到本機時虛機重名，將原虛機暫時重新命名為migrating-domain_name 3. 發送設定檔 4. 真正遷移是在 forkHelper(cmd, fd, saveInputHandler, False)中做的，即建立一個子進程來執行xc_save，而主進程繼續執行後面的步驟: 其中，cmd為 /usr/lib64/xen/bin/xc_save fd domid 0 0 str(int(live)|(int(hvm)<<2)) （xc_save的參數格式為
/usr/lib64/xen/bin/xc_save iofd domid maxit maxf flags） def saveInputHandler(line, tochild):
log.debug("In saveInputHandler %s", line)
if line == "suspend":
log.debug("Suspending %d ...", dominfo.getDomid())
dominfo.shutdown('suspend')
dominfo.waitForSuspend()
if line in ('suspend', 'suspended'):
dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP2,
domain_name)
log.info("Domain %d suspended.", dominfo.getDomid())
dominfo.migrateDevices(network, dst, DEV_MIGRATE_STEP3,
domain_name)
if hvm:
dominfo.image.saveDeviceModel()
if line == "suspend":
tochild.write("done\n")
tochild.flush()
log.debug('Written done') 5. 發送qemu裝置的狀態(即/var/lib/xen/qemu-save.7) 6. 由於傳入的checkpoint參數為True，將suspended的虛機resume 7. 最後，將重新命名的虛機改為原來的名字下面是restore部分tools/xen/xend/server/SrvDaemon.py
xend在啟動的時候會開啟8002連接埠的監聽：
Daemon()-->run()-->relocate.listenRelocation()-->tcp.TCPListener(RelocationProtocol, port, interface = interface, hosts_allow = hosts_allow)
相應的參數在xend設定檔中設定：(xend-relocation-server yes)(xend-relocation-port 8002)(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')建立串連後的處理由RelocationProtocol完成，其中最重要的函數為op_receive: def op_receive(self, name, _):
if self.transport:
self.send_reply(["ready", name])
try:
XendDomain.instance().domain_restore_fd(
self.transport.sock.fileno(), relocating=True)
except:
self.send_error()
self.close()
else:
log.error(name + ": no transport")
raise XendError(name + ": no transport") 而XendDomain.domain_restore_fd()是通過
XendCheckpoint.restore(self, fd, paused=paused, relocating=relocating)來完成的。tools/python/xen/xend/XendCheckPoint.py def restore(xd, fd, dominfo = None, paused = False, relocating = False): 1. 首先接收並驗證SIGNATURE 2. 接收設定檔 3. 確保本機上沒有與要遷移的虛機同名或同UUID的虛機 4. 根據接收的設定檔建立XendDomainInfo結構： dominfo = xd.restore_(vmconfig)-->XendDomainInfo.restore(config) 5. 如果原虛機設定了node_to_cpu，則綁定vcpu到相應的pcpu 6. 建立image並分配記憶體（shadow_memory + memory_dynamic_max） restore_image = image.create(dominfo, dominfo.info) balloon.free(memory + shadow, dominfo) 7. 真正遷移是在forkHelper(cmd, fd, handler.handler, True)中做的，即建立一個子進程來執行xc_restore，而主進程繼續執行後面的步驟:
其中，cmd為: cmd = map(str, [xen.util.auxbin.pathTo(XC_RESTORE),
fd, dominfo.getDomid(),
store_port, console_port, int(is_hvm), pae, apic])
（xc_save的參數格式為/usr/lib64/xen/bin/xc_restore iofd domid store_evtchn console_evtchn hvm pae apic）
def handler(self, line, _):
m = re.match(r"^(store-mfn) (\d+)$", line)
if m:
self.store_mfn = int(m.group(2))
else:
m = re.match(r"^(console-mfn) (\d+)$", line)
if m:
self.console_mfn = int(m.group(2))
5. 接收qemu裝置的狀態 (即/var/lib/xen/qemu-save.7) 6. 設定虛機cpuid，並建立虛機：
restore_image.setCpuid()
os.read(fd, 1) # Wait for source to close connection
dominfo.completeRestore(handler.store_mfn, handler.console_mfn)：
self._introduceDomain() self.image = image.create(self, self.info) if self.image: self.image.createDeviceModel(True) self._storeDomDetails() self._registerWatches() self.refreshShutdown()
7. 最後，啟動虛機：
dominfo.waitForDevices() dominfo.unpause()
xc_save和xc_restore的源碼 暫時還沒有仔細看。
參考：http://wiki.prgmr.com/mediawiki/index.php/Chapter_9:_Xen_Migration
