Experience Design Technology
Immersive Web Case Study
ML-enabled immersive web experience to showcase the case study and interactive storytelling
- Studio
- frog design / individual project
- Year
- 2023
- Responsibility
Led the project, collaborated with a team of design technologists at frog SF studio, implemented ML classes (body segmentation, face recognition, etc.), and texture mapping of canvas rendering into 3D geometry, and backend for test environment.
Technology
This web experience is built in a React application, leveraging React Three Fiber to represent APIs of Three.js. In the 3D model of the gallery room, box geometries are located where interactive experiences take place. For each box geometry, a canvas data from P5.js interactive sketch is mapped. Each interactive experience is implemented as separate P5 sketches, where they integrate with computer vision features: TensorFlow.js provides open source models available such as body segmentation, or face tracking, and we implemented custom classes to take web camera input and return body segmentation data or face feature points tracking.
