Our team has been involved in developing a cutting edge solution for a remote controlled stereoscopic 360° camera.
The solution developed includes two parts: an iOS client app for displaying video, controlling camera settings and a server part running on Microsoft Windows platform that performs color filtering (color correction, white balance etc) for panorama stitching.

iOS Client Application

The iOS app was developed by our team from scratch. It has MVC architecture with React Native front-end and C/Objective-C back-end including FFmpeg(libav) RTSP.h264 Video Player and Google Virtual Reality SDK for displaying video stream in stereoscopic mode.

Server Part

The Server part was developed with C/C++ 11 and uses OpenCV(3.3.0), Live555, FFmpeg(libav), RTSP.h 264 video streamer, HTTP server. The core operation part was based on PPL(Parallel Patterns Library for multi-cure CPU), CUDA, NVidia VRWorks and does an ultra-fast speed optimized image pre/post filter processing (like demosaic, white balance, gamma/color correction, anti vignetting, chromatic aberration); fast but accurate stitching of mono/stereo panorama images based on optical flow (and other) algorithms.

Challenges & Achievements

#Speed. The developed solution had been provide an algorithm that creates panoramas with a minimum 25 fps; with a resolution 1024x512 px and without significant artifacts. A detailed analysis of possible algorithms of qualitative panorama generation has shown the possibilities for parallelizing the inseparable operation cycles due to the preliminary search for positions in these cycles, where data does not "moving" between iterations. Due to this, such cycles were separated and executed parallelly with GPU/CUDA. Additionally, we developed own memory manager for frames that allowed to save a lot of valuable time for allocating/freeing the necessary space and moving data between CPU and GPU. All stated above allow to meet specification requirements.

#Quality. The was an issue of artifacts reducing (as on the one frame, as when transitions between frames on video sequence). The algorithm had been able to generate panorama with the maximum quality of cross-linking even if the content of the cross-linked images went into a logical conflict each other. At the same time the size of the resulting panorama should 8400x4096 px. To resolve this issue we use Dense approaches (Functional Minimization in Pixel Flow), OpenCV Contributive.

Conclusion

Our team received an excellent experience building this multi-platform solution. Stereoscopic 360° cameras is a cutting edge technology that changes things around video content. We continue develop our expertise in this young, growing and complex subject.

Let’s discuss the next great project!