1% Migration
2% Revision 2
3
4\clearpage
5
6# Basics
7--------------- -------------
8        Status: **Supported**
9
10  Architecture: x86
11
12     Component: Toolstack
13--------------- -------------
14
15# Overview
16
17Migration is a mechanism to move a virtual machine while the VM is
18running.  Live migration moves a running virtual machine between two
19physical servers, but the same mechanism can be used for non-live
20migration (pause and copy) and suspend/resume from disk.
21
22# User details
23
24No hardware requirements, although hypervisor logdirty support is
25required for live migration.
26
27From the command line, `xl migrate/save/restore` are the top level
28interactions.  e.g.
29
30    xl create my-vm.cfg
31    xl migrate my-vm localhost
32
33or
34
35    xl create my-vm.cfg
36    xl save my-vm /path/to/save/file
37    xl restore /path/to/save/file
38
39Xen 4.6 sees the introduction of Migration v2.  There is no change for
40people using `xl`, although the `libxl` API has had an extension.
41
42# Technical details
43
44Migration is formed of several layers.  `libxc` is responsible for the
45contents of the VM (ram, vcpus, etc) and the live migration loop, while
46`libxl` is responsible for items such as emulator state.
47
48The format of the migration v2 stream is specified in two documents, and
49is architecture neutral.  Compatibility with legacy streams is
50maintained via the `convert-legacy-stream` script which transforms a
51legacy stream into a migration v2 stream.
52
53* Documents
54    * `docs/specs/libxc-migration-stream.pandoc`
55    * `docs/specs/libxl-migration-stream.pandoc`
56* `libxc`
57    * `tools/libxc/xc_sr_*.[hc]`
58* `libxl`
59    * `tools/libxl/libxl_stream_{read,write}.c`
60    * `tools/libxl/libxl_convert_callout.c`
61* Scripts
62    * `tools/python/xen/migration/*.py`
63    * `tools/python/scripts/convert-legacy-stream`
64    * `tools/python/scripts/verify-stream-v2`
65
66## libxl
67
68With migration v2 support, LIBXL_HAVE_SRM_V2 and LIBXL_HAVE_SRM_V1
69are introduced to indicate support.  `domain_restore_params` gains a new
70parameter, `stream_version`, which is used to distinguish between legacy and
71v2 migration streams, and hence whether legacy conversion is required.
72
73# Limitations
74
75Hypervisor logdirty support is incompatible with hardware passthrough,
76as IOMMU faults cannot be used to track writes.
77
78While not a bug in migration specifically, VMs are very sensitive to
79changes in cpuid information, and cpuid levelling support currently has
80its issues.  Extreme care should be taken when migrating VMs between
81non-identical CPUs until the cpuid levelling improvements are complete.
82
83# Testing
84
85Changes in libxc should be tested with every guest type (32bit PV, 64bit
86PV, HVM), while changes in libxl should test HVM guests with both
87qemu-traditional and qemu-upstream.
88
89In general, testing can be done on a single host using `xl
90save/restore` or `xl migrate $VM localhost`.
91
92Any changes to the conversion script should be tested in all upgrade
93scenarios, which will involve starting with VMs from Xen 4.5
94
95# Areas for improvement
96
97* Arm support
98* Live looping parameters
99
100# Known issues
101
102* x86 HVM guest physmap operations (not reflected in logdirty bitmap)
103* x86 HVM with PoD pages (attempts to map cause PoD allocations)
104* x86 HVM with nested-virt (no relevant information included in the
105  stream)
106* x86 PV ballooning (P2M marked dirty, target frame not marked)
107* x86 PV P2M structure changes (not noticed, stale mappings used) for
108  guests not using the linear p2m layout
109
110# References
111
112Xen Developer Summit 2015 Presentation
113[video](https://www.youtube.com/watch?v=RwiDeG21lrc) and
114[slides](http://events.linuxfoundation.org/sites/events/files/slides/migv2.pdf)
115for Migration v2
116
117# History
118
119------------------------------------------------------------------------
120Date       Revision Version  Notes
121---------- -------- -------- -------------------------------------------
1222015-10-24 1        Xen 4.6  Document written
1232015-12-11 2        Xen 4.7  Support of linear p2m list
124---------- -------- -------- -------------------------------------------
125