Sharing & Recording Screen In 3D Virtual Office

INTRODUCTION

Recently, we created a room where people can interact and call each other. You can find the link here: Realtime Multiplayer Worlds with ThreeJS, Socket.IO, and WebRTC . However, to transform it into a fully functional virtual office, we need to implement a function to share and record the screen. This enhancement will greatly enrich online meetings, making them more comprehensive and enjoyable for all participants.

In this article, we'll explain how we added these features and how they work in detail. 

You can check our demo in this link: https://udl-3d-web-multiplayer.onrender.com/ 

FEATURES

In this version, users can share their screens, similar to other meeting platforms like Google Meet. Users can choose to share a chrome tab, window or entire screen.

Users can also record the shared screen. The recorded video will contain both the video and audio of the shared screen, as well as the voices of all the participants. A preview video will be displayed on popup in the corner of the screen after you finish the recording. User can choose to save or discard the video by press buttons below the video.

HOW IT WORKS

Sending the screen stream

Previously, we utilized WebRTC to establish a Peer Connection between two participants, enabling the sending and receiving of the webcam stream through that connection.

To implement the screen sharing feature, we need to find a way to send an additional stream - the screen sharing stream - to another person. We can't simply create another connection because it's too resource-intensive; doubling the current connections would lead to a large number of connections, which could easily overwhelm the web platform.

So we decided to add a sharing stream to the existing connection and then using socket to signal all other participants to update their connection. It significantly reduce performance overhead.

Record the sharing screen.

To record the sharing screen, we utilized RecordRTC. This library allows us to record a MediaStream (video + audio). However, in reality, a user receives multiple streams in a meeting; for example, each person's webcam stream is a separated stream, and we may have multiple participants in a meeting.

Our goal is to record the video and audio of the sharing screen, along with all the voices of other participants, as we need to hear people's explanations.

We achieve this goal by merging all the audio streams into one audio stream and creating a stream with the sharing screen video and the merged audio stream.

Merged stream for recording.

Display the screen on 3D space.

ThreeJS enables us to use a video as a texture and apply it to meshes or models in 3D space. Initially, we search for the video component containing the data of the sharing stream. Then, we extract this video data to create a VideoTexture. Subsequently, we generate a material from this texture and apply it to the 3D model of the television.  Also make sure you set the colorSpace to “srgb” to that new texture. Otherwise, the color may appear altered.

CONCLUSION

With the addition of screen sharing and recording capabilities, the virtual office has transformed into a fully functional meeting space.

In the future, we will continue to add more features to make this an even more enjoyable experience for online meetings.