With this project, Microsoft Research is exploring how machine learning can help to solve some of the problems of image loss associated with putting cameras behind the screen, which can help frame remote conversations in a more realistic space setting. The two most significant problems associated with camera-in-display technology today are of perspective (for making eye contact in video conferencing) and the diffraction caused by OLED screens.

Using transparent OLED (T-OLED) displays, we can put a camera behind the screen, potentially solving the issue of perspective. Yet since the glass is not fully clear, it degrades image quality by adding diffraction and noise when looking through it. To address this, Microsoft used a U-Net neural-network structure to compensate for the image loss inherent in photographing through a T-OLED screen, which both increase the signal-to-noise ratio and de-blurs the image.

On the issue of perspective, both the arrangement of the participants relative to each other and their distance (proxemics) are meaningful aspects of nonverbal communication but are not taken into account by current videoconferencing systems. The solution to this, according to Microsoft Research, lies in first identifying the speakers using image segmentation achieved with a custom CNN model. It then becomes possible to scale the incoming video after identifying the speaker in the remote view, so that the remote participant can appear on the local display in a lifelike image. One way to accomplish this would be to zoom the whole image and re-center it on the speaker. However, for this project, the researchers went a step further. They extracted and scaled the individual independently of the background, as seen in the image shown above.

Their research concludes, "Human interaction in videoconferences can be made more natural by correcting gaze, scale, and position by using convolutional neural network segmentation together with cameras embedded in a partially transparent display. The diffraction and noise resulting from placing the camera behind the screen can be effectively removed using U-Net neural network. Segmentation of live video also makes it possible to combine the speaker with a choice of background content."