Hongxin(April) Fu

SHOW: All Art&Design Digital Humanities Data Visulization Others

Desciption

“Geometric Gestures: Pattern Interaction” is an innovative project that explores the synthesis of traditional geometric patterns with modern interactive technology. This initiative aims to bridge the gap between historical art forms and contemporary digital interfaces by allowing users to interact directly with geometric designs through intuitive gestures.
We extract patterns from the Forbidden City’s real scenes and abstract them into geometric shapes. Then, we design gestures to mimic these geometric shapes, allowing users to trigger introductions to the patterns by making the corresponding gestures.
This project utilizes MediaPipe for its machine learning capabilities, enabling real-time gesture recognition and interaction within a web development framework. This allows users to dynamically interact with geometric patterns via simple hand gestures, directly in their web browser.

Gesture Design

We extract patterns from real scenes and transform them into geometric shapes. Then, we design gestures that mimic these shapes.

Interaction Flow

The audience can follow the on-screen instructions to perform gestures. When the correct gesture is detected, the introductory animation will be triggered.

Technical implementation process

We mannually collect the photos of gestures as our dataset. Then we use mediapipe the train our model in google colab and export the model file to be called in the web development. Additionally, we have configured the program to restart automatically if no gestures are detected for 30 seconds.

Refection

This project holds significant meaning for me as I conceptualized, developed, presented, and deployed it. When tasked with creating a work related to the patterns of the Forbidden City, I wanted to do something engaging. I was inspired by an animation created by my classmate, who is also the co-author of this work. I aimed to connect people with these beautiful, silent patterns. I wondered how people could interact with them and feel their essence. This led me to consider gestures as a means for people to understand the shapes of the patterns, while an accompanying animation could narrate their stories.

Extension

Later, we expand this project in another comptetition where we mainly focused on the patterns on the windows in the forbidden city. This project combines interactive screens and AIGC technology, allowing users to explore and create personalized art pieces inspired by the intricate patterns of Forbidden City window designs. Users begin by selecting a pattern they wish to explore. The system provides details about the pattern’s characteristics and approximate location. Guided by these hints, users navigate a hand-drawn map of the Forbidden City to locate the palace housing the chosen pattern. By clicking on the palace name, they access real-life images of the palace and search for the exact position of the pattern within the image. Once the pattern is identified, the system displays detailed information, including its name, high-definition images, historical background, and cultural significance. Finally, users mimic the shape of the pattern with hand gestures to trigger the generation of a personalized artistic image.
For the image generation, we use ControlNet’s Canny model to extract the outline of the selected window pattern, and then combine it with the user’s photo captured by the camera to create a personalized image. To reduce waiting time, the system starts capturing the photo and generating the image as soon as the user begins the gesture recognition phase. This ensures that by the time the user completes the interaction, the final image is ready for viewing.

(GAELDH 2024, Hong Kong)

May you find yourself among countless choices.