Tangoed with RTCPeerConnection

Recently I had done a task with RTCPeerConnection to create a peer-to-peer room where the pair can communicate with each other via video and audio, share their screen and share a white board together.

Components

Peers

  1. Browser 2, aka the creator of the room
  2. Browser 1, aka the one who joins the room after the creator

Running server instance for data communication (Bridge Server)

The instance will do the following things in video chat and screen sharing:

  1. receive the SDP offer from Browser 1 and send it to Browser 2
  2. receive the SDP answer from Browser 2 and send it to Browser 1
  3. receive the Ice Candidate from Browser 1 and send it to Browser 2
  4. receive the Ice Candidate from Browser 2 and send it to Browser 1

The instance will do the following things in whiteboard sharing:
1. send the dot position whenever the dot is touched by either browser, and send it to the other one
2. receive the final board when either browser finishes the painting, and send it to the other one
3. merge all changes into the room board, for future use

Frontend App

for UI.

Ice Servers

for video and audio stream communication.
I used those public google ones.

Processes

Video Chat

  1. SDP and Ice Candidate communication, see MDN for more. The Bridge Server as is described before is responsible for this. In this part an alternative is scaledrone.
  2. Browser 1 sends the SDP offer to Browser 2, meanwhile adding its own stream into the RTCPeerConnection
  3. Browser 2 receives the offer and sends the created answer to Browser 1
  4. Browser 1 receives the answer and sends the generated ice candidate to Browser 2
  5. Browser 2 receives and processes the candidate, then sends the generated ice candidate to Browser 1
  6. Browser 1 receives and processes the candidate
  7. Now chat.

Screen Sharing

  1. Both browsers choose a screen to share.
  2. SDP and Ice Candidate communication, see MDN for more. The Bridge Server as is described before is responsible for this. In this part an alternative is scaledrone.
  3. Browser 1 sends the SDP offer to Browser 2, meanwhile adding its own stream into the RTCPeerConnection
  4. Browser 2 receives the offer and sends the created answer to Browser 1
  5. Browser 1 receives the answer and sends the generated ice candidate to Browser 2
  6. Browser 2 receives and processes the candidate, then sends the generated ice candidate to Browser 1
  7. Browser 1 receives and processes the candidate
  8. Now start the screen sharing.

Whiteboard Sharing

Both browsers has access to the whiteboard first, enabling them to edit and view the board. In this part I have to implement my own server.

  1. Either browser can start drawing on the whiteboard by touching or mousedown-moving
  2. When drawing, each point will be sent to the server.
  3. The server will share the point to another browser if there r 2 browsers connecting to this room.
  4. In the other browser, the points will become black, indicating that browser is doing drawing.
  5. When the drawing is done, the b64 image of the final board will be sent to the server.
  6. The server will share the image to another browser if there r 2 browsers connecting to this room, meanwhile merging it with the room board for future use.
  7. In the other browser, the b64 will be drawn to its whiteboard.
  8. Whenever either browser quits and comes again, its white board will be restored using the room board.

Difficulties encountered

  1. Incremental changes for server room whiteboard? I've actually tried node-canvas package. For a single canvas, it performed really well. But when I wanted to merge a canvas to the room canvas by drawImage using the merged image, it always failed. The room canvas always returned an empty image when I used toDataURL. So I use merge-images package instead and only store b64 image for server room whiteboard.
  2. 2 paths in a canvas at the same time? I didn't find a way to do so, that's why I implemented black dot on the browser when it received the points from another browser. I think I should've implemented 2 canvas instead. Possible now. Solved by making use of 2 canvas.