# Migration at Scale — 120 GB Media Library

Honest assessment of running `r2-offload` against a large WordPress install (~100–200k attachments, 120 GB).

---

## TL;DR

Architecturally yes, it will work. Practically: run the migration via WP-CLI over a weekend, with `screen`/`tmux`, and stage it (scan → migrate → verify → prune). Don't drive it from a browser tab.

---

## What already scales

- **Chunked, resumable scanner** — cursor persisted in `r2_offload_scan_state` option. Handles 200k+ attachments without reloading the world.
- **Indexed items table** — `UNIQUE(attachment_id)`, `KEY(status)`. `claim_pending` is a narrow, indexed query.
- **Resume-safe** — crash, redeploy, reboot: already-offloaded rows stay offloaded; only `pending`/`failed` get retried.
- **Streaming uploads** — `google/cloud-storage` SDK uses resumable uploads for files > 5 MB. PHP memory stays flat; no full-file buffering.
- **Byte-preserving** — zero re-encoding. `.webp` / `.avif` sidecars picked up automatically.
- **Rate limiter + exponential backoff w/ jitter** — on 408 / 429 / 5xx. Settings: `rate_ms`, `max_retries`, `backoff_base_ms`. Defaults conservative (150 ms between GCS calls).
- **Log rotation** — file log at `logs/offload.log` rotates at 5 MB; DB log capped at 10k rows.

---

## What to tune before a 120 GB run

### Plugin settings
- **Rate limiting** → `rate_ms = 50` (still well under GCS 1k writes/s per bucket).
- **Max retries** → 4 is fine.
- **Auto-prune** → **OFF** for the initial migration pass. Prune only after a full `verify --all` pass.
- **Serve source** → **WordPress (local)** until verification completes. Flip to **Google Cloud (remote)** after verify passes.

### Server (for CLI-driven runs)
- `php.ini`: `memory_limit = 512M` (CLI), `max_execution_time = 0` (CLI only).
- Adequate disk during migration (local files still present until prune).
- Outbound bandwidth matters more than anything else — expect wall-clock time dominated by upload, not plugin overhead.

---

## Rough throughput math

- 120 GB ÷ ~800 KB avg object (original + 4 derivative sizes + webp sidecar) ≈ **~150k objects**.
- Single-process, sequential, `rate_ms=50`: ≈ 2–4 s per attachment end-to-end.
- Est. total: **40–60 hours worst case**. Actual bottleneck = upload bandwidth.
- At 200 Mbps sustained: 120 GB ≈ ~80 min of pure transfer; realistic total ≈ **8–14 hours** with overhead + throttling.

---

## Recommended procedure

### 1. Pre-flight
```bash
wp media-offload gcs test-connection
wp media-offload gcs scan --dry-run
```
Review bytes + count. Estimate GCS storage + egress cost at this scale before committing.

### 2. Scan (builds the queue)
```bash
wp media-offload gcs scan
```

### 3. Migrate (long-running; keep alive)
```bash
screen -S offload
cd /path/to/wp-root
wp media-offload gcs migrate
# Ctrl-A D  → detach
# screen -r offload  → reattach later
```
If it dies / host reboots: re-run the same command. Already-offloaded rows untouched; only `pending`/`failed` re-enter the queue.

### 4. Verify
```bash
wp media-offload gcs verify --all
```
Only proceed to cutover after this is clean.

### 5. Cutover
```bash
wp media-offload gcs serve remote
```
Or flip the switch in Admin → Migration → "Serve media from".

### 6. Prune (staged, in batches)
```bash
wp media-offload gcs prune --dry-run --limit=500
wp media-offload gcs prune --limit=500
# repeat; never prune everything in one call
```

### 7. (Optional) restore any attachment back to local
```bash
wp media-offload gcs restore --attachment-id=<id>
```

---

## Without SSH (Rapyd.cloud / similar managed hosts)

Rapyd.cloud does not expose SSH. That removes the recommended path. Options, ranked best-first:

### Option A — Ask Rapyd to run WP-CLI
Most managed WP hosts without shell *do* support WP-CLI internally — either via their control panel or by having support run one-off commands. Open a ticket:

> Please run the following WP-CLI commands against our site:
> ```
> wp media-offload gcs scan
> wp media-offload gcs migrate
> wp media-offload gcs verify --all
> ```
> They're long-running; please run inside `screen`/`tmux` or your equivalent. The plugin is idempotent; re-running is safe.

If they say no: move to Option B.

### Option B — Install Action Scheduler + drive from admin
Install the [Action Scheduler](https://wordpress.org/plugins/action-scheduler/) plugin. The existing `Scheduler` class already detects it (`as_enqueue_async_action`) and will route jobs through AS's durable background runner. Benefits:

- Survives admin tab closing (AS runs on WP-Cron / on every front-end request).
- No browser polling needed for the work itself — only for progress display.
- Jobs persist in AS's own table with retry/backoff.

Trade-off: AS advances on traffic. Low-traffic sites need either real visitor traffic or an **external cron hitting `/wp-cron.php` every minute** to keep the queue moving. Managed hosts like Rapyd usually run system cron correctly — verify this with them.

With AS installed, the existing Start button in Admin → Migration is a valid full-migration driver. Leave the tab open to watch progress; the migration continues even if you close it.

### Option C — Migrate from a local mirror, point at the same bucket
1. Pull the production uploads directory locally (via host's backup / (S)FTP / export tool).
2. Stand up a local WP using the same DB + uploads.
3. Run the migration locally against the real production GCS bucket (SSH available locally).
4. After verify: on production, install the plugin with the same bucket credentials, run `wp media-offload gcs scan` (or just import the generated `wp_r2_offload_items` table rows via SQL) so production's DB knows the remote mapping.
5. Flip serve-source to remote on production. Prune locally whenever.

This is the most bulletproof route: production never sees migration load.

### Option D — Browser-driven, long-lived tab
Acceptable if Options A–C are off the table. Open Admin → Migration → Start, keep a dedicated always-on machine with the tab open. AJAX tick loop will drain at whatever pace the browser + AJAX endpoint sustain. Each tick is a separate HTTP request → PHP timeouts are per-attachment, not per-migration. If the tab closes: re-open and click Start, state resumes.

Trade-off: fragile (browser update, OS reboot, Wi-Fi hiccup can stall it). Good enough for 10–20 GB; risky for 120 GB.

---

## Known gaps (Phase 4 candidates)

- No true parallel workers. Single-threaded. 2–4× throughput possible with AS + worker count > 1.
- No per-run ETA in CLI. Use `tail -f logs/offload.log` for throughput estimation.
- No checksum verification beyond size match. MD5/ETag comparison would catch silent corruption.
- No bucket lifecycle templating (cold-tier rules for old objects).
- No adaptive rate tuning based on observed 429 rate.

---

## Cost checklist (before pulling the trigger)

- **Storage**: 120 GB × GCS class rate × months. Standard ~$0.020/GB/mo. ~$2.40/mo.
- **Writes (Class A)**: ~150k × $0.005 / 1000 = **~$0.75 one-time**.
- **Reads (Class B / CDN)**: depends on traffic. Cache aggressively with Cloud CDN.
- **Egress**: first GB free + region-dependent rates. For CDN-fronted public sites, egress to the CDN is often the main line item.
- **Dry-run scan first** to avoid over-estimating on orphaned / missing files.

---

## Pre-cutover safety checklist

- [ ] Full backup of production (DB + uploads) before first migrate run.
- [ ] Service account has **Storage Object Admin** scoped to the bucket only (not project-wide).
- [ ] UBLA-compatible setup confirmed (legacy ACL OFF unless fine-grained bucket).
- [ ] `allUsers` → `Storage Object Viewer` granted on bucket IF serving directly from `storage.googleapis.com`. Otherwise Cloud CDN with signed URLs.
- [ ] `scan --dry-run` output reviewed; bytes total matches expectations.
- [ ] Subset migration tested (Admin or CLI with `--limit=100`) — images load from remote and `srcset` rewrites correctly.
- [ ] `verify --all` passes clean before pruning.
- [ ] Prune in batches; monitor free disk.
- [ ] Rollback plan documented: flip `serve remote` → `serve local`, `restore --attachment-id=X` for any issue.
