PliARs
Prototype of Force Feedback Tool for Mixed Reality Applications
I. Gonsher and Z. Lei, "Prototype of Force Feedback Tool for Mixed Reality Applications," 2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2021, pp. 508-509, doi: 10.1109/ISMAR-Adjunct54149.2021.00123.
Context
Physical tools are dedicated to physical objects in physical environments.
Virtual tools are dedicated to virtual objects in virtual environments.
Virtual tools are dedicated to virtual objects in virtual environments.
What is the new paradigm that allows one users to move fluently between the virtual and the physical world?
Ideation
The tool integrates the pliers as a functional part and the phone as an interaction input. To achieve an ergonomic composition, we tested out different compositions regarding the relative position of the two parts, such as a phone on the front/backside and horizontal/vertical orientation.
It allows the user to both feel and sees in both modalities. This allows users to see the virtual object and manipulate it accordingly; it also offers opportunities for other affordances, including collaboration at a distance through video conferencing, for example. There is also potential to use these kinds of tools in scenarios where collaboration and instruction occur at a distance.
Prototyping
The 3d-printed casing houses the servo motor. This motor simulates the physical resistance of clamping on objects. It also operates the opening of the pair of pliers as if they were bounced back by springs.
Screen Wireframe
We designed two affordances: the instructor mode and the apprentice mode.
In both modes the user can use the phone to control the tool’s functionality while seeing through the phone’s camera of the actual pair of pliers.
In both modes the user can use the phone to control the tool’s functionality while seeing through the phone’s camera of the actual pair of pliers.
Implementation
The prototype uses an Arduino Bluetooth LE Nano board to control the servo motor. It allows the tool to communicate with the Unity interface on the attached smartphone wirelessly via Bluetooth.
The analog feedback from the servo motor provides positional data of the pliers’ movement. This movement is mapped in the virtual interface in real-time.
In turn, this virtual model determines when the motor should provide force feedback by locking its position.
Buttons on the exterior of the handle mirror the same functionality of the virtual controls, making elements of the interface easily discoverable.
The analog feedback from the servo motor provides positional data of the pliers’ movement. This movement is mapped in the virtual interface in real-time.
In turn, this virtual model determines when the motor should provide force feedback by locking its position.
Buttons on the exterior of the handle mirror the same functionality of the virtual controls, making elements of the interface easily discoverable.
Scenario 1: Instructor records manipulations
Scenario 2: Apprentice follows to recreates manipulations
AR Rehabilitation
Ergonomic Hand Motion Assistance and AR Rehabilitation
This paper was published and presented at the IEEE VR 2024 Workshop: IDEATExR.
Memory Palace
I recently started to think that if something doesn’t exist in the digital world, it doesn’t exist at all. Ironically, digital content becomes transient when the file format is not supported, the webserver is down, or even when the hardware fails. So maybe it’s better to store those contents in our human memory than in rapidly accessed memories?
![]()
I set off to involve memory episodes in a replicable, controllable digital premise. When remembering an event we experienced in the past, not only do we experience being the subject of the conscious episode, but we also experience being the protagonist in the memory scene. [1] The human body’s action becomes a critical part of memory.
![]()
Charles Augustus Magnussen was a character from the tv series Sherlock. He knows all the secrets of every important person and blackmails them with that information. At his base, Appledore, Magnussen stores all these secrets in a vault and looks at them occasionally. As it turns out, the “vault” is his brain, and he retrieves files by meditating in a small room.
![]()
This method is called the “method of loci,” or memory palace. The memory palace established a connection between the memory content and the place that holds it. As a result, memories can easily be recalled when the person comes back later to visit the palace.
![]()
I made a game that allows me to build my own memory palace by putting images in rooms, like a museum. I can put any photo on the wall, either in a frame or a window. I can add new rooms infinitely. Rooms are connected with virtual portals, through which I can move back and forth between rooms. These rooms are generated with random shapes for the pictures to go onto. They function as creases in our brain. The bigger surface it has, the more information it can store. So this memory palace augments the existence of some physical experiences.
![]()
This is the memory palace of my last two semesters at RISD. Images on my iCloud account enumerated it. There are interesting places and events, foods, animals, screenshots, etc.
I set off to involve memory episodes in a replicable, controllable digital premise. When remembering an event we experienced in the past, not only do we experience being the subject of the conscious episode, but we also experience being the protagonist in the memory scene. [1] The human body’s action becomes a critical part of memory.
[1] Lin Ying-Tung, Visual Perspectives in Episodic Memory and the Sense of Self
Charles Augustus Magnussen was a character from the tv series Sherlock. He knows all the secrets of every important person and blackmails them with that information. At his base, Appledore, Magnussen stores all these secrets in a vault and looks at them occasionally. As it turns out, the “vault” is his brain, and he retrieves files by meditating in a small room.
This method is called the “method of loci,” or memory palace. The memory palace established a connection between the memory content and the place that holds it. As a result, memories can easily be recalled when the person comes back later to visit the palace.
I made a game that allows me to build my own memory palace by putting images in rooms, like a museum. I can put any photo on the wall, either in a frame or a window. I can add new rooms infinitely. Rooms are connected with virtual portals, through which I can move back and forth between rooms. These rooms are generated with random shapes for the pictures to go onto. They function as creases in our brain. The bigger surface it has, the more information it can store. So this memory palace augments the existence of some physical experiences.
This is the memory palace of my last two semesters at RISD. Images on my iCloud account enumerated it. There are interesting places and events, foods, animals, screenshots, etc.
Palace - RISD
Palace - Panoramas
Transpositional Dioramas
You are visiting two dioramas that float in the middle of emptiness. They respectively feature the exterior and interior view of the same bay window.
This whale shuttles you back and forth between two dioramas.
After each run, it goes inside the dioramas and looks at you from behind the glass.
This whale shuttles you back and forth between two dioramas.
After each run, it goes inside the dioramas and looks at you from behind the glass.
Exterior Diorama:
This is a bay window copied from the Carr House at RISD, a Queen Anne style house built at 1885. During the period, whales had been the powering source of modern civilizations. The majority of them had been hunted for whale products. The fractional surviving species are still struggling not to extinct.
This is a bay window copied from the Carr House at RISD, a Queen Anne style house built at 1885. During the period, whales had been the powering source of modern civilizations. The majority of them had been hunted for whale products. The fractional surviving species are still struggling not to extinct.
Interior Diorama:
This is the inside of the Carr House bay window. It is a library decorated with whale products:
1. a corset made with whale baleen,
2. an umbrella made from whalebone,
3. a pipe made from whalebone,
4. a candle and lipstick made from whale oil,
5. a doll whale.
This is the inside of the Carr House bay window. It is a library decorated with whale products:
1. a corset made with whale baleen,
2. an umbrella made from whalebone,
3. a pipe made from whalebone,
4. a candle and lipstick made from whale oil,
5. a doll whale.
Now, what do you see? The window, yes. And a collection of whale products that are burning.
Whales had been the powering source of modern civilization for hundreds of years. The fractional surviving species are still struggling not to extinct.
This particular whale shuttles you between the two dioramas. Then it goes to the backside and looks at you from behind the glass.
Whales had been the powering source of modern civilization for hundreds of years. The fractional surviving species are still struggling not to extinct.
This particular whale shuttles you between the two dioramas. Then it goes to the backside and looks at you from behind the glass.
Interactive Diorama
These composite views are animated using OpenCV and a pre-trained face-recognition model, which allows for an interactive view with masks on.
The interactive view utilizes parallax effect and operates on pure 2D images (as listed above) to render a three-dimensional visual experience of visiting the dioramas.
The interactive view utilizes parallax effect and operates on pure 2D images (as listed above) to render a three-dimensional visual experience of visiting the dioramas.
This is a part of the Transpositional Dioramas Project.
see here