XR Foundation: the AI layer under PICO 4

Context

VR live streaming is brutal on compute: binocular, ultra-high-resolution frames; effects that have to follow a moving performer; and a headset running on a mobile chip. Every effect and beautification step had to fit in real time.

How it works

The AI XR SDK sits between the models and the headset apps. It splits work between cloud and device and spends compute only where the viewer will notice. The hardest problems became patent filings:

• Key-point-driven effect rendering for VR livestreamsLocates target key points in ultra-high-resolution VR livestream frames and concentrates effect rendering on the related regions, cutting computation and latency.
• Cloud / headset split rendering for VR livestream effectsChooses where each effect runs based on how real-time it must be: rendered entirely in the cloud, or detected in the cloud and rendered on the VR headset.
• Region-based processing of 8K VR livestream framesSelects only the regions of a binocular, ultra-high-resolution VR frame that need processing, such as beautification, so 8K livestreams stay real-time.
• Distortion-aware beautification for binocular VR livestreamsUndistorts the target region of a binocular VR frame, applies beautification or filters, then maps the result back into the original projection, improving quality while reducing computation.
• Perspective windows across virtual scenesOpens a portal-like window inside the current VR scene that shows a view into another pre-built scene, computed from the viewer's observation point.
• Compositing real objects with virtual goods in VR / ARShows a real object, such as the user's foot, alongside virtual goods; the selected item is composited onto it, as in a virtual try-on, and can then be associated with the user.

Several of these inventions are in the product I led and built on top of the AI XR SDK.

What I owned

• Product lead for the XR Foundation: led a 70+ person team building AI-enhanced XR SDKs and services for PICO.
• Built the product that calls the AI XR SDK, including mixed-reality live-streaming interactions.
• Co-inventor on six patent families from this work, across VR live rendering, spatial interaction, and immersive commerce.

Coverage & related