iPhone Duo animation: reconstruction notes

These notes describe our reconstruction of the reference render, not its original implementation. The camera calibration and 37-frame comparison were updated on September 21. Earlier experiments are labeled separately below.

Small rendering errors can break continuity

We built the prototype with Three.js for the hardware and camera, plus a custom WebGL2 shader for the screen image. A browser renderer captures repeatable poses. Swift and AVFoundation extract timestamped reference frames and encode the output; Python and OpenCV handle alignment and image comparisons.

Several implementation details affected the result:

For an intermediate blur strength, the shader samples the two neighboring blur levels and blends their colors. This approximates an intermediate Gaussian. Each level retains the full image resolution. The array alone uses about 77.3 MB, before temporary canvases and GPU overhead, so this setup still needs optimization for a mobile app.

High-contrast text exposed a subtler problem when we substituted an Empirical screenshot. Our rays initially came from interpolated positions on each display surface. Tiny numerical differences caused overlapping surfaces to disagree by up to three channel levels out of 255.

We changed the shader to derive the ray directly from the output pixel and inverse camera projection. Both surfaces then used the same calculation. The maximum difference fell to zero across the 54 GPU sample pairs we checked, before brightness adjustment.

Timing needed similar care. We estimated fold angles at reference timestamps, including roughly 90° at 3.95 seconds and fully open at nine seconds. A smooth, monotone curve connects those checkpoints. Retiming changes when a pose appears without changing how it’s drawn; 14 desktop and mobile comparisons produced byte-identical images of equivalent poses.

Our closing loop reverses that opening. We didn’t fit the later closing movements in the source, and our blur and shading depend on angle rather than movement direction.

Camera calibration: September 21

The original 10-degree camera flattened the moving leaf, and the slow animation inherited framing from an earlier, faster clip. We fitted a 32.33-degree perspective camera and smooth pose trajectory to 19 silhouettes across the first opening. This is an approximate geometric fit, not recovery of the source renderer’s camera.

We also moved the reference image center toward the hinge during the handoff, removed a lateral mask that cut into the perspective-enlarged leaf, restored the camera apertures, and brightened the stationary inner display near edge-on. The blur formula and its fourteen levels stayed unchanged.

Both versions were freshly rendered at 37 identical decoded timestamps. Time-weighted aligned foreground-union RGB error fell from 65.86 to 30.18 channel levels; silhouette intersection-over-union rose from 0.7693 to 0.9750. The 18 quarter-second samples excluded from the silhouette fit improved similarly: 66.41 to 30.67 RGB error and 0.7680 to 0.9731 silhouette overlap. These frames had been visible during our work, so this is not a blind evaluation.

This comparison uses Pillow/SciPy masks and resampling. It is separate from the historical 271-frame OpenCV benchmark below, and measures uncompressed frames. Per-frame results and provenance are available. Desktop/mobile checks retained zero channel difference across 54 overlapping reference-image GPU sample pairs; the app variant passed 66 pairs.

Wallpaper crop, baked clock/controls, hinge details, and reflective materials remain approximate. The app video in the article retains its earlier camera.

Historical baseline: what still differed

Our surfaces agree with one another more closely than the complete reconstruction agrees with the reference. The camera perspective, screen boundaries, clock placement, landscape crop, brightness, and hardware appearance still differ.

We measured the full first opening at 271 requested times, representing 265 distinct decoded reference frames. After uniformly scaling and translating the rendered device to align it with the reference, the time-weighted silhouette intersection-over-union was 0.769. That compares the area occupied by both silhouettes with the area occupied by either one.

Average RGB error over the union of the devices was 65.96 channel levels on the 0–255 scale. That region includes mismatched outlines. Measuring only where both devices overlap gives a lower error, but leaves those outline differences out.

These are baseline measurements of the full opening. The earlier crop experiment and the GPU sampling checks answer narrower questions. Neither establishes that we’ve recovered the original renderer. The calibration above addresses the camera and image handoff; image crop, controls, shading, and hardware remain areas for refinement.

Technical appendix: our reconstruction settings

The equations and measurements below describe our implementation. Angles in code use radians: θ = 0 is fully open and θ = π is fully closed. W is one panel’s width; d is normalized distance from the hinge.

Image projection and focus

For camera position C and visible screen point P, expressed in device coordinates, the equivalent ray-plane calculation is:

r = P − C
t = (zimage − Cz) / rz
Q = C + t × r

Q determines the source-image coordinates. The implemented shader constructs the ray from the fragment pixel and inverse camera projection to avoid per-surface interpolation differences.

The separate focus intersection uses the rotating leaf’s midplane, ignoring shell thickness. It rejects intersections behind the camera or beyond the leaf, and guards nearly parallel rays.

For the original lock-screen rendering mode, the layout bounds and image center are:

left = −W × max(0, cos(θ))
right = W
center = (left + right) / 2

The image retains its scale and aspect ratio. The calibrated reference instead uses center = W × max(0, −cos(θ)) / 2, with full-width bounds [-W, W] and physical meshes providing lateral clipping. The app version fixes the image center at the hinge.

Blur and tone

The circular Gaussian kernel in source-image coordinates is:

Gσ(x, y) ∝ exp(−(x² + y²) / (2σ²))
σ = 96 × sin(θ) × d³

Canvas generates fourteen blur levels once in memory:

0, 0.5, 1, 2, 3.5, 5, 8, 12, 18, 26, 38, 54, 74, 96

We extend the image edges by ceil(4 × 96) + 2 = 386 pixels on each side before convolution. The shader uses linear filtering within each array layer, blends neighboring layers, and flips the vertical coordinate to account for Canvas row order. A blend of two Gaussian-filtered images isn’t generally identical to a single Gaussian at the interpolated strength.

At 90°, the model’s maximum σ is 96 source-image pixels at the free edge, which is nearly edge-on. It doesn’t imply a 96-pixel blur across the output frame.

Brightness multiplies the blurred encoded RGB values:

stationary inner gain = 1 − 0.65 × (1 − max(0, cos(θ)))
cover gain = 1 − 0.35 × sin²(θ)
rotating inner gain = stationary inner gain × (1 − 0.5 × sin(θ) × d)

In the calibrated reference, stationary inner gain becomes 0.5 + 0.5 × cos(θ) for angles up to 90°, and 0.5 + 0.2 × cos(θ) beyond 90°. The rotating panel uses that gain with the same edge darkening. The original app tone remains unchanged.

The renderer uses NoColorSpace, raw shader output, and disabled tone mapping. These are appearance controls, with no fitted black offset, reflection texture, exposure compensation, or temporal settling in the current model.

Validation results

Historical experiments used different regions and rendering paths. Their errors shouldn’t be combined into one improvement curve:

Those recorded results predate the current shared-image renderer. The earlier material benefited from pixel-center corrections, more low-blur levels, and direct convolution for small kernels. Timestamp exclusion was retrospective: frames had already been inspected, and each supplied its own alignment. The tone-removed comparison was an ablation, not a separately optimized competitor. Original temporary per-frame reports weren’t available when the reconstruction notes were written.

A synthetic test used a known eight-pixel Gaussian, gain 0.8, and offset 18 to check recovery. A separate flat shader control using the same OpenCV blur array on CPU and GPU measured 0.164 mean channel error and a maximum of 1. Those controls test the analysis and implementation, not the final animation’s resemblance to the video.

For the historical baseline’s full 0–9-second opening, the time-weighted measurements were:

MAE is mean absolute channel error; RMSE weights large errors more heavily; PSNR expresses error relative to the signal range. The reference fits within 960×720 without stretching. Foreground alignment uses uniform scale and translation, without rotation, perspective warping, independent width scaling, or color correction. A silhouette threshold of 20 channel levels from the median border color defines the masks. Per-frame errors are averaged with trapezoidal time weights.

These measurements exclude encoder-added endpoint holds and our final MP4 compression. They include differences unrelated to blur, and no model settings were optimized against this first full-opening score.

Reproducing the preview

The prototype retains earlier experiments, so the final combination needs explicit switches:

index.html?projection=shared&focus=travel&tone=fold&motion=slow&camera=calibrated

Add &ui=app for the Empirical screenshot. Omit camera=calibrated to reproduce the earlier camera used in the article’s app video. The prototype includes fit-camera.py, compare-calibration.py, and test-calibrated-camera.cjs for reproducing the calibration and checks. The export samples 271 frames over nine seconds at 30 fps, with short endpoint holds added during encoding. The encoder needs an explicit speed of 1; its historical default is quarter speed. Reference extraction records actual decoded timestamps so repeated frames retain their duration in comparisons.