Earlier work
Projects from the Ph.D., NYU ITP and before
Before my current work on AI agents, I spent about ten years on extended reality, personal devices and interactive art. This page collects that work: Ph.D. research at the University of Toronto, industry research, projects from NYU's Interactive Telecommunications Program (ITP), and earlier work in physics and design at Peking University and Tsinghua University.
Ph.D. research, University of Toronto
2018 to 2025Advised by Tovi Grossman. The thesis, Exploring Bidirectional Interfaces and Precise Input with Personal Devices in Extended Reality, asked how the phone in your pocket and the headset on your face can work together, and how to make selection in VR precise.
PinchLens
A free-hand technique for picking small, densely packed targets in VR. When you start to pinch, a magnifying bubble enlarges nearby targets, hand motion is scaled down for precision, and a cursor shows the estimated pinch point.
With Ludwig Sidenmark, Mauricio Sousa and Tovi Grossman.
Beyond the Phone
Phone apps in a headset that move between mirrored, magnified and augmented views. The prototype reads an app's interface layout at runtime to add controls and spatial views. Done during my time at Google Labs.
With Xun Qian, David Kalmar, Mehdi Tayarani, Eric J. Gonzalez, Mar Gonzalez-Franco, David Kim and Ruofei Du.
A later collaboration
- 2026Prop-Chromeleon: Adaptive Haptic Props in Mixed Reality through Generative Artificial IntelligenceH. Wang, F. Zhu, B. Huang, Z. Wang, L. Sidenmark. ACM DIS 2026. Honorable Mention Award.
Industry research
2017 to 2024Google Labs
Cross-device interaction between phones and headsets, with Ruofei Du and David Kim. The work led to Beyond the Phone (IEEE VR 2025) and a US patent application on choosing a headset's input method from the handheld device's input method.
NVIDIA Research
Designed and deployed ARSIM, a simulator for optical see-through augmented reality displays. Worked with Josef Spjut, Morgan McGuire, David Luebke and Turner Whitted.
Holojam Inc.
Co-located multi-person VR and a mixed-reality system that lets spectators watch a VR performer. This grew out of Ken Perlin's lab at NYU and led to two US patents with him (11,635,627 and 11,836,285).
NYU Interactive Telecommunications Program
2015 to 2017A two-year master's at Tisch School of the Arts, spent on physical computing, illusions, games, performance and, more and more, shared VR and AR. Much of the later work was done with Ken Perlin's group. Descriptions follow my project blog from the time (itpzhufy.com, now offline; copies survive in the Internet Archive).
Chalktalk VR/AR
Ken Perlin's Chalktalk brought into shared VR and AR: people draw, and the sketches become interactive, animated objects that everyone in the space can see and use. A way to talk in VR beyond avatars and voice. The research was sponsored by Oculus for classroom use.
Ken Perlin, Zhenyi He and Fengyuan Zhu (CreMedia 2018).
PhyShare
Haptics for VR from small actuated robots. Because users cannot see the robots, one robot can stand in for different virtual objects, either as something to touch or as an invisible carrier of a real object, and remote collaborators can share the same physical feedback.
With Zhenyi He and Ken Perlin.
Holojam
I was one of the main contributors to Holojam, an engine for co-located multiplayer VR used in a series of shared experiences. It also ran with Vive and with trackers we built ourselves, including a camera-based one.
HoloKit
A low-cost, cardboard-style mixed reality headset built around a phone, with inside-out tracking, gesture detection and multiplayer, and a wider field of view than HoloLens. Open source, and demoed at SIGGRAPH 2017.
Supernova
A live performance combining motion capture, projection onto a self-built screen environment, and illusions. The scene was built in Unreal and tracked with OptiTrack, and the show was performed at MAGNET in New York. Made in the Bodies in Motion class.
LAUNCH
An immersive film for the IAC video wall in New York, 120 feet wide and 11 feet high, at 11520 by 1080 pixels. Made in Mimi Yin's Big Screens class.
By Quest Kennelly and Fengyuan Zhu.
BoomChaCha!
A physical, rhythm-based role-playing game. Three players, a warrior, a defender and a mage, share one life bar and fight monsters with handheld weapons, doing best when they move to the beat together.
F. Zhu, W. Sun, C. Zhang, R. Ricks (CHI 2016 Extended Abstracts).
Brainwave Minions
A row of singing boxes driven by an OpenBCI headset. Blinking makes one side sing; concentrating produces an alpha wave that makes all of them sing together.
Group project; video by Hao Tseng.
Corridor in the Wall
A perspective illusion that makes viewers believe a corridor runs inside the wall. The image shifts with the viewer's position as they move.
More ITP projects
Mad Table
An illusion built with face detection, from the question: what if the objects around us had a spirit of their own and did not want us to know?
Cardboard AR
A cardboard augmented reality viewer that holds a phone below a Pepper's ghost reflector, made after a tabletop Pepper's ghost experiment.
One Punch
A punching game made at a hackathon: a Myo armband measures the speed of your punch, and an Arduino scoreboard and an on-screen circle show the score.
Decentralized Network vs Artificial Intelligence
After reading Kevin Kelly's Out of Control: a game where the audience, each holding a red paper, steers one character together, competing against a robot arm controlled by a program.
Robotic hand
A servo-driven robotic hand with three joints per finger, built to make human gestures as a medium between people and machines.
Class projects and sketches
- 2017JS canvas libraryNetworked Media midterm: a Node, Express and MongoDB site for uploading and hosting canvas sketches.
- 2016Vive tracker boardIntro to PCBs: assembled an open-source HTC Vive lighthouse sensor board by hand, from stencil to reflow.
- 2016Reinvent the WheelA remote-controlled robot car with an xBee radio and long-exposure light trails, a coin-weighted flywheel, a phenakistoscope, a walking linkage and a pull-toy dog.
- 2016Save the States! and Catch the ObamaArcade games with wireless motion controllers and a fabricated arcade box.
- 2016Musicar, Obama made in China and Obama's eyeReadymades and projection mapping: a toy car combined with a recorder so it can be played as an instrument, built from ITP junk-shelf parts, a motion-triggered souvenir figure, and a projected eye in an acrylic glass.
- 2015Visualizing natureVisual Language: a color composition in HSB space modeled on Kepler's laws, with ten thousand particles orbiting a center and slowly losing energy.
- 2015These DaysVideo and Sound final with KT, Becca and Naoki.
- 2015ITPer's SmileA first short video, shot on a GoPro.
- 2017Monocular Visual-Inertial State Estimation for Mobile Augmented RealityP. Li, T. Qin, B. Hu, F. Zhu, S. Shen. IEEE ISMAR 2017.
Tsinghua x-studio and Peking University
Until 2015I studied physics at Peking University, with a second degree in art, and worked as a research assistant in x-studio, Department of Information Art and Design, Tsinghua University, on tangible interfaces and persuasive technology.
Persuasive Screen
A computer screen that senses the user's sitting posture and physically moves to prompt a correction while they type. I designed the hardware and software and ran the experiment.
With Ke Fang and Xiaojuan Ma.
S.T.A.R. (Shooting Star)
A multiplayer game on a Microsoft PixelSense table, based on Breakout: one team lights tangible stars and the other darkens them, with a virtual comet and physical black holes and spacecraft.
With Xu Tongda, Luo Yang, Wang Anqi, Zhang Cunjun and Zhang Fengjie. My role: product management, lead software design and hardware design.
T.E.S.T.
A tangible kit for learning circuits: the parts stay physical while the wiring and analysis move into software, so beginners can experiment safely. Tested with primary school students.
Same team as S.T.A.R.
PolyHinge
Shape-changing tangible robots based on the rule that a polygon can be cut and rotated into another polygon, explored for position and gesture detection, AR and tangible control, and games.
Elastic Space
Balls rolling on a stretched membrane are the classic picture of curved spacetime. We asked whether such a system really behaves like gravity, and whether an apparent gravitational constant can be defined and measured in it.
Walking on the Earth
My first photo album: landscape photographs from three years, including the West Lake after snow and Kanas in the morning.
Also from these years
- 2013The Transformation of Li QunA film made for the centenary of the School of Physics, Peking University, about an undergraduate's journey through physics.
- 2013If You Are the OneMy first film as director, the story of a student's search for love.
- Annual concert of the Peking University String AssociationCommittee member, organizer and performer.
- LEGO2NANOAn open-source, low-cost atomic force microscope for schools, built by students from Tsinghua, Peking University and University College London. I wrote the Python control GUI.
- Quantum Mechanics Demonstration LabA Java program that shows particles in a box and the harmonic oscillator, with adjustable parameters.
Open-source code
Selected public repositories on GitHub (zhufyaxel). Forks, class templates and empty repositories are left out.
Oculus Quest 2 Unity template
A Unity project with Oculus Integration and the settings needed to run Quest 2 apps both on the headset and in the editor, written up step by step for people new to Unity.
TimeLapseToolkit
Scripts for time-lapse photography: batch image compression, and turning image sequences into videos and GIFs with FFmpeg.
Smaller repositories
- 2019CSC428 Assignment 2: extra help documentationA guide I wrote as a teaching assistant at the University of Toronto, collecting students' common questions about the assignment code.
- 2017Nature of Code evolutionsFinal project for Nature of Code at ITP: a 3D evolution system in Unity in the spirit of Daniel Shiffman's steering examples.
- 2017Google Tango SDK workshopUnity project and sample Android builds from a Tango SDK workshop at the ITP Unconference.
- 2015 to 2016BoomChaCha game codeProcessing and Arduino code for BoomChaCha, the rhythm game listed under NYU ITP.