Let it flow: Recreating Water Flow for Virtual Reality

The physical laws of everyday water flow were established two centuries ago. However, scientists today struggle to simulate disrupted water flow virtually, e.g., when a hand or object alters its flow.

Now, we have harnessed the power of deep reinforcement learning to replicate the flow of water when disturbed. Replicating this agitated liquid motion, as it is known, allowed them to recreate water flow in real time based on only a small amount of data from real water. The technology opens up the possibility for virtual reality interactions involving water.

Crucial to the breakthrough was creating both a flow measurement technique and a flow reconstruction method that replicated agitated liquid motion.

To collect flow data, we placed buoys embedded with special magnetic markers on water. The movement of each buoy could then be tracked using a magnetic motion capture system. Yet this was only half of the process. The crucial step involved finding an innovative solution to recovering the detailed water motion from the movement of a few buoys. We overcame this by combining a fluid simulation with deep reinforcement learning to perform the recovery.

Reinforcement learning is the trial-and-error process through which learning takes place. A computer performs actions, receives feedback (reward or punishment) from its environment, and then adjusts its future actions to maximize its total rewards over time, much like a dog associates treats with good behavior. Deep reinforcement learning combines reinforcement learning with deep neural networks to solve complex problems.
First, we used a computer to simulate calm liquid. Then, we made each buoy act like a force that pushes the simulated liquid, making it flow like real liquid. The computer then refines the way of pushing via deep reinforcement learning.

Previous techniques had typically tracked tiny particles suspended inside the liquid with cameras. But it still remained difficult to measure 3D flow in real-time, especially when the liquid was in an opaque container or was opaque itself. Thanks to the developed magnetic motion capture and flow reconstruction technique, real-time 3D flow measurement is now possible.

This technology will make VR more immersive and improve online communication. It will enable the creation of VR games where you can control things using water and actually feel the water in the game. We may be able to transmit the movement of water over the internet in real time so that even those far away can experience the same lifelike water motion.

The research results were published in the online bulletin of the ACM Transactions on Graphics, the scientific journal of the Association for Computing Machinery (ACM), on 17 September 2023, and were presented orally at the top conference on computer graphics and interactive technology top conference SIGGRAPH Asia 2023 (12-15 December, Sydney, Australia).