Live Migration

Overview

Live Migration allows you to move a running virtual machine (instance) from one hypervisor server to another with minimal downtime (typically less than 1 second). Your instance continues running during the migration process, ensuring uninterrupted service to your applications and users.

Zero-Downtime Moves

Live migration is perfect for load balancing, planned maintenance, and hardware upgrades without service interruption.

How It Works

During live migration:

1. Your instance continues to run on the source server
2. Memory and CPU state are transferred to the destination server
3. Storage is either accessed from shared storage or transferred
4. A brief pause occurs (< 1 second) during final synchronization
5. Your instance resumes on the destination server
6. Network connectivity is maintained

Migration Types

Shared Storage Migration (Recommended)

Best for: Instances with disks on shared/network storage (datastores)

• Speed: Fast (5-10 minutes for typical VMs)
• Downtime: < 1 second
• What's transferred: Memory and CPU state only
• Requirements: Destination server must have access to the same datastore

Recommended

This is the fastest and most reliable migration method. Use shared storage (datastores) whenever possible.

Block Migration

Best for: Instances with disks on local storage

• Speed: Slower (depends on disk size - 15-60+ minutes)
• Downtime: < 1 second
• What's transferred: Memory, CPU state, AND disk contents
• Requirements: Destination server must have sufficient free storage

Slower Process

Block migration copies all disk data during migration. A 100GB disk may take 30-60 minutes depending on network speed.

Server Requirements

Network Requirements

Both hypervisor servers must meet these network requirements:

1. Bidirectional Connectivity

   • Source and destination servers must reach each other over the network
   • No NAT or firewall blocking between servers

2. Required Open Ports

Port(s)	Protocol	Direction	Purpose
22 (or custom SSH port)	TCP	Bidirectional	SSH communication for migration orchestration, ensure the port is mentioned in the hypervisor configuration page as well
16509	TCP	Destination → Source	Libvirt remote API (optional, for monitoring)
49152-49215	TCP	Bidirectional	Migration data transfer (dynamic port range)

Firewall Configuration

Ensure your firewall allows traffic on these ports between hypervisor servers. You may need to configure iptables, firewalld, or your network firewall.

Example Firewall Rules (iptables):

# On both source and destination servers
# Allow SSH
iptables -A INPUT -p tcp --dport 22 -s <other-server-ip> -j ACCEPT

# Allow libvirt
iptables -A INPUT -p tcp --dport 16509 -s <other-server-ip> -j ACCEPT

# Allow migration data transfer
iptables -A INPUT -p tcp --dport 49152:49215 -s <other-server-ip> -j ACCEPT

3. Network Bandwidth
   • Minimum: 1 Gbps link recommended
   • Optimal: 10 Gbps for large instances or multiple simultaneous migrations
   • Latency: < 10ms preferred (same datacenter/network)

Hardware Requirements

1. CPU Compatibility
   • Same CPU vendor (Intel to Intel, AMD to AMD)
   • Compatible CPU generations
   • Similar CPU features (SSE, AVX, etc.)

CPU Mismatch

Migrating between incompatible CPUs may cause the instance to crash or fail. Always verify CPU compatibility first.

2. Memory

   • Destination server must have sufficient free RAM
   • Formula: Free RAM ≥ Instance RAM + 2GB buffer

3. Storage (for block migration)

   • Destination must have free storage ≥ total disk size
   • Same or faster storage performance recommended

Software Requirements

Both servers must have:

• Same or compatible Linux distribution
• Libvirt installed and running
• QEMU/KVM installed
• SSH server running
• Compatible versions:
  • Libvirt 7.0+ recommended
  • QEMU 5.0+ recommended

Version Compatibility

While minor version differences are usually fine, staying within 1-2 major versions ensures best compatibility.

Network Configuration Requirements

Option 1: Keep Same IP Addresses

• Destination server must have access to the same network subnets
• Network must support IP mobility (same L2 network or proper routing)
• Best for: Servers in same datacenter/network segment

Performing Live Migration

Step 1: Verify Prerequisites

Before initiating migration, verify:

✅ Server Connectivity

• SSH connectivity between servers works
• Required firewall ports are open
• Network latency is acceptable

✅ Resources Available

• Destination has sufficient RAM
• Destination has sufficient storage (if block migration)
• Destination CPU is compatible

✅ Storage Setup (for shared storage migration)

• Destination server can access the datastore
• Storage paths are correctly configured

✅ Instance State

• Instance is running (live migration requires running VM)
• No ongoing operations on the instance

Step 2: Initiate Migration

1. Navigate to Instances

   • Go to Instances in the admin panel
   • Locate the instance you want to migrate

2. Open Migration Dialog

   • Click on the instance
   • Select Actions → Live Migrate

3. Configure Migration

   Destination Server:

   • Select target hypervisor from dropdown

   IP Address Configuration:

   • ✅ Keep IP Addresses: Maintains current IPs (requires same network subnets)
   • ⬜ Assign New IPs: Allocates new IPs from destination subnets

   Storage Mapping:

   • Map each disk to destination storage
   • System auto-detects shared storage

   Advanced Options:

   • Auto-Converge: ✅ Recommended - Helps busy VMs complete migration
   • Compression: ✅ Recommended - Reduces network bandwidth (~50-70%)
   • Bandwidth Limit: Optional - Limit migration speed (MB/s)
   • Post-Copy: ⬜ Experimental - Not recommended

Post-Copy Warning

Post-Copy mode starts the VM on destination before all memory is transferred. If network fails during post-copy, the VM will be lost! Only use for critical zero-downtime scenarios.

4. Review and Confirm
   • Review migration summary
   • Verify source and destination details
   • Click Start Live Migration

Step 3: Monitor Progress

The migration progress screen shows:

• Overall Progress: 0-100% completion
• Migration Phase: Current operation stage
• Memory Transfer: Memory copied vs remaining
• Disk Transfer: Disk blocks copied (if block migration)
• Estimated Time: Remaining duration
• Live Migration Stats:
  • Iteration count (memory sync cycles)
  • Transfer speed
  • Downtime accumulation

Real-Time Updates

Progress updates every 10 seconds. The UI shows live statistics during migration.

Step 4: Completion

When migration completes:

• ✅ Instance is running on destination server
• ✅ Network connectivity restored
• ✅ Storage accessible
• ✅ All configurations preserved
• ⚠️ Source server cleaned up (if configured)

Advanced Configuration

Migration Performance Tuning

Auto-Converge

What it does: Throttles the VM when memory changes faster than it can be transferred

When to enable:

• ✅ Busy databases or high-memory applications
• ✅ Long-running migrations that don't seem to converge
• ❌ Lightly-loaded VMs (unnecessary)

Side effects: May slightly reduce VM performance during migration

Compression

What it does: Compresses memory pages during transfer using XBZRLE algorithm

Benefits:

• Reduces network bandwidth by 50-70%
• Faster migration on bandwidth-constrained links
• Minimal CPU overhead

When to enable:

• ✅ Always recommended unless you have 10+ Gbps dedicated link
• ✅ Cross-datacenter migrations
• ✅ Network with limited bandwidth

Bandwidth Limit

What it does: Limits migration network traffic to specified MB/s

When to use:

• Shared network with production traffic
• Avoid saturating network links
• Controlled migration speed for predictability

Recommended values:

• 1 Gbps link: 100-125 MB/s
• 10 Gbps link: 500-1000 MB/s
• Unlimited: Leave blank (maximum speed)

Post-Copy Mode

What it does: Starts VM on destination before all memory is transferred

Characteristics:

• ⚠️ Experimental feature
• ⚠️ Minimal downtime but risky
• ⚠️ VM lost if network fails during post-copy

High Risk

If network connection is lost during post-copy migration, your VM cannot be recovered! Only use when zero-downtime is absolutely critical.

When to consider:

• VM with very high memory churn that won't converge
• Absolutely critical zero-downtime requirement
• Reliable, dedicated network between servers

Using Private Network

If your servers have dedicated private network interfaces:

1. Configure private IP addresses on both servers
2. Enable Use Private Network option during migration
3. Migration traffic uses private interface, keeping public network free

Benefits:

• Faster migration speeds (dedicated bandwidth)
• No impact on production traffic
• Better security (isolated network)

Troubleshooting

Common Issues

Migration Stalls / Won't Complete

Symptoms:

• Progress stuck at 60-80%
• Memory percentage not increasing
• Time remaining keeps increasing

Causes:

• VM has very high memory write rate
• Application is too busy

Solutions:

1. ✅ Enable Auto-Converge option
2. ⚠️ Consider Post-Copy (risky)
3. 🔄 Retry during lower activity period
4. 📉 Temporarily reduce VM workload if possible

Network Connection Failed

Symptoms:

• Migration fails immediately
• Error: "Cannot connect to destination"

Causes:

• Firewall blocking required ports
• SSH connectivity issue
• Network routing problem

Solutions:

1. Verify firewall rules allow ports 22, 16509, 49152-49215
2. Test SSH connectivity: ssh root@destination-ip
3. Check network routing between servers
4. Verify destination server is reachable

Insufficient Storage

Symptoms:

• Migration fails during disk transfer
• Error: "No space left on device"

Causes:

• Destination storage full
• Storage mapping incorrect

Solutions:

1. Verify destination storage has sufficient free space
2. Check storage mapping configuration
3. Clean up old backups or snapshots
4. Use different storage pool with more space

CPU Incompatibility

Symptoms:

• Instance crashes after migration
• Error: "Unsupported CPU feature"

Causes:

• Source and destination have incompatible CPUs
• Destination CPU missing required features

Solutions:

1. Verify both servers have compatible CPU families
2. Check CPU features match: lscpu | grep Flags
3. Consider using CPU compatibility mode (contact support)
4. Migrate to server with compatible CPU

Instance Won't Start on Destination

Symptoms:

• Migration completes but instance doesn't start
• Error in migration logs

Causes:

• Storage path incorrect
• Network configuration mismatch
• Missing dependencies

Solutions:

1. Check migration logs for specific errors
2. Verify storage accessibility on destination
3. Confirm network configuration matches
4. Contact support with migration ID

Migration Status Indicators

Status	Meaning	What to Do
Pending	Migration queued	Wait for processing
Preparing	Setting up migration	Normal - system preparing
Migrating	Active migration in progress	Monitor progress
Completing	Final synchronization	Almost done - wait
Completed	✅ Success	Verify instance is running
Failed	❌ Error occurred	Check error message, review logs

Getting Help

If you encounter issues:

1. Check Migration Logs

   • View detailed logs in migration details
   • Look for specific error messages

2. Review Instance Status

   • Verify instance state on source and destination
   • Check current running location

3. Contact Support

   • Provide migration ID
   • Include error messages
   • Describe what was happening when issue occurred

Expected Downtime

• Typical: 200-500 milliseconds
• Best case: < 100 milliseconds
• Worst case: 1-2 seconds

Service Continuity

Most applications experience no noticeable interruption. TCP connections typically survive the brief pause.

FAQ

Can I migrate a stopped instance?

No, live migration requires the instance to be running. For stopped instances, use regular (cold) migration instead.

Can I migrate across datacenters?

Yes, but consider:

• Requires block migration (slower)
• Higher network latency may impact performance
• IP addresses will likely change
• Ensure adequate bandwidth between datacenters

What happens if migration fails?

The instance remains running on the source server. You can:

• Review error messages
• Fix the underlying issue
• Retry the migration

Can I roll back a migration?

Live migration is one-way. However, you can:

• Migrate back to original server using another live migration
• The instance data is preserved on both ends until cleanup

Does live migration work with all operating systems?

Yes, live migration is OS-agnostic. It works with:

• All Linux distributions
• Windows Server
• BSD variants
• Any OS supported by KVM

Can I migrate multiple instances simultaneously?

Yes, but consider:

• Available network bandwidth
• Source and destination server resources
• Performance impact on running instances
• Recommended: 2-3 concurrent migrations maximum

How secure is live migration?

Migration is secure:

• ✅ SSH encrypted connections
• ✅ Temporary SSH keys (auto-deleted)
• ✅ No data exposed to public network
• ✅ Private network option for isolation