Virtual Reality (VR) has recently caught the attention of the press, inspired the creativity of a multitude of content producers, and captivated the imaginations of the public. Many companies are entering the VR arena and are throwing their hats in the ring with their own Head-Mounted Displays (HMDs), cameras, hardware, and software for the consumer market.
While the biggest market to date for VR has been in the gaming world, there are emerging opportunities in other arenas that can benefit from the use of 360° VR video. Potential VR applications abound in several verticals, including training, education, real estate, travel, cinema, medical, enterprise, entertainment military and law enforcement, security, therapy, and more.
Since VR takes an interactive, immersive sensory experience to a whole new level, it is literally a new form of communication with its own language, user interfaces, and technical requirements. It is a new storytelling and experiential paradigm that can virtually bring people into live events and locations so they can view the events as if they were actually there.
Impressive as VR seems, the process of producing high quality recorded 360° video content is difficult, time consuming, and frustrating since there are no set standards. VR production is fraught with quirks, bugs, and workarounds in almost every step of the production process.
Each manufacturer throwing its hat in the VR ring is still developing its own development systems and delivery mechanisms, and some are still finalizing the production workflows and ecosystems. Companies such as Oculus VR, HTC, and Sony will be launching their final consumer hardware and platforms in 2016, which in effect may prove a catalyst for widespread adoption of VR and usher in its mainstream acceptance.
Live Streaming 360° VR Video
How likely are we to see VR streaming move into the mainstream in the coming months or years? Live streaming alone presents plenty of challenges for most producers, since there are so many moving parts. VR production is even more difficult. The fusion of the two can be a technical nightmare that requires a lot of trial and error and patience. If you can overcome the frustration and stay on course, it can truly lead to the creation of a new viewer experience that’s both exciting and groundbreaking.
VR’s clunky production process and unpredictable workflow hasn’t stopped creators, producers, software developers and companies from experimenting and exploring new methods of creating VR content. The process of shooting recorded VR is starting to gain a lot of traction. Live streaming panoramic video goes a step further by taking the core concept of VR 360° and streaming it out live to a mobile app or website that can be viewed with HMD in conjunction with a mobile phone or computer.
Introducing Equirectangular Video
They main objective in VR video is to output an equirectangular, spherical projection stream. Much like the construction of panoramic photographs, equirectangular video takes video from several camera sources and “stitches” them into a single sphere. Then it maps that sphere to a flat plane, like a flattened map of the Earth.
Once the video is stitched and flattened, you can stream that single video output to an RTMP server which then can be viewed through headsets, mobile devices, and desktops.
Components of a VR Streaming Workflow
The VR streaming workflow has several key components. In order to start producing and live streaming 360° VR, you will need the following:
• a complete camera rig
• a high-end graphics workstation with a powerful graphics card that can ingest and stitch in real time
• stitching software to show you what your shot looks like on the fly so you can make adjustments accordingly
• an encoder to send it to a CDN
• a final delivery platform.
DIY 360º Cameras Rigs
When it comes to camera systems, you can either assemble a Do-It-Yourself rig or purchase a manufactured one. One thing you will realize is that many of the manufactured rigs have either not yet been released and are extremely expensive. So if you want to start capturing and streaming live VR video now at a lower cost, you need to look into putting together a DIY rig.
One of more popular DIY camera solutions for 360º VR is combining multiple GoPro 4 Black cameras into a single rig. Typically, these rigs are based on arrays that hold multiple GoPro cameras in 6, 10, 12 and 14-camera configurations. The manufacturers of these harnesses include Freedom360 and Hero360.
While Freedom360 mounts (Figure 1, below) start as low as $499, complete DIY rigs can cost around $3,000−$5,000, since you need to first purchase the array, the GoPros, necessary cables, USB power hubs, media, tripods, and other accessories.
Figure 1. The Freedom360 DIY VR camera rig. Click the image to see it at full size.
Unfortunately, the main issue with GoPro cameras is that they tend to overheat, which generally limits your recording time. They perform poorly in hot environments and can sometimes shut down after several minutes of use. Operating GoPros in close proximity to each other on a VR mount compounds this problem. You can overcome these issues if you keep cameras properly ventilated, and adding fans does help.
Manufactured 360º Camera Rigs
There are more than 30 different types of VR camera systems either available no or releasing soon. They come in all different shapes and sizes and, as you might expect, each manufacturer touts its offering as the best solution out there. These cameras are manufactured by major companies, by startups, and even as Kickstarter-funded projects.
Since there are too many products to mention, I’ll cover the ones that strike me as the most interesting. Ultimately, you’ll need to weigh your options in light of your projects and expectations and choose the system that would work best for you, whether you decide to purchase or rent a turnkey solution, or build one yourself.
GoPro Odyssey and Six-Camera Rig
Understanding the applicability of its popular HERO action/POV cameras to VR applications, GoPro recently announced the availability of the Odyssey (Figure 2, below), with an MSRP of $15,000. The Odyssey consists of 16 GoPro HERO4 Blacks, 16 Array Bacpacs, 16 Micro SD cards, a microphone, power cables and accessories, and Pelican carrying case. It also features GenLock camera syncing, multicamera group settings control, and extended batteries.
Figure 2. GoPro’s $15,000 16-camera Odyssey VR rig. Click the image to see it at full size.
Odyssey was created to streamline the VR content production process for Google’s Jump platform, which puts content creation and consumption under one umbrella. After a video is captured, the platform automatically stitches videos into a single panoramic VR video. GoPro is selling the device only to “select creators” who apply via its website. GoPro will be releasing a simplified six-camera rig as well, but pricing and details have not been released yet.
IM360
Immersive Media is one of the more high-profile VR production companies. Immersive Media’s IM360 is responsible for providing the technology behind the VR stream of the red carpet event at the MTV Video Music Awards.
Immersive Media is offering solutions that are based on their custom packages, which consist of a pairing a camera and streaming encoder for approximately $60,000.
Samsung Project Beyond
Samsung’s Project Beyond (Figure 3, below) is a saucer-like camera that captures 360-degree images and video footage that can viewed on their Gear VR headset. It has 17 HD cameras and can capture 3D still panoramas, record video, and stream live video with built-in high speed connectivity.
Figure 3. Samsung’s 17-camera, streaming-ready Project Beyond. Click the image to see it at full size.
Project Beyond also features fully automated stitching technology.
Nokia OZO
The OZO (Figure 4, below) is Nokia’s first foray into commercial VR. It’s targeted toward the professional content creator market. The orb-like camera captures stereoscopic 3D video with eight synchronized global shutter sensors and spatial audio with eight microphones.
Figure 4. The orb-like Nokia OZO. Click the image to see it at full size.
Nokia will provide various tools that will help producers create videos with a built-in playback system. It will be compatible with various headsets like the Oculus Rift as well. OZO is slated to ship in the fourth quarter of 2015.
Radiant Images
Radiant Images is pushing the limit in VR cameras with a variety of intriguing-looking systems. The company’s offerings range from cameras that rent from $895 per day (including two VR 360° models in an underwater housing) to other systems that cost nearly 10 times as much. Radiant is focusing on the high end of the market; the big-budget film industry is one of their primary markets.
The company’s most compelling system is the Mobius head-mounted POV camera, which rents for about $7,500 per day and has a brace built in to enable steady first-person shots. One drawback is that Radiant’s cameras are available only for rental, but the rental includes at least one technical support staff to help you configure and operate the units.
Hardware
So those are the key players developing solutions specifically geared to VR production and streaming. What else will you need to get into the VR game?
VR requires a lot of processing power, and having high-end GPU is key. You can choose to build your own workstation or purchase a unit from a manufacturer.
If you’re building your own rig, I would recommend a system with maximum RAM, a high-powered graphics card with at least an NVIDIA GeForce GTX 980 GPU, and an Intel processor such as a Core i7 Intel 6700 or above.
Few companies are currently building high-end workstations that are specifically configured for VR production. But one company, Shinobi Systems (full disclosure, I am a Shinobi Systems shareholder) does build turnkey systems that can handle the entire VR streaming process. These workstations are able to take multiple direct camera inputs from 8, 16, 24, and 32 camera sources such as GoPros, stitch them in real time, and encode directly out to a CDN via RTMP stream.
360 Video-Stitching Software
After you’ve decided on what camera to use, stitching software is a critical component for streaming. Most of the software out there is designed to stitch video in postproduction.
But VideoStitch, a French company has developed real-time auto stitching software for video, called Vahana VR. Vahana VR (Figure 5, below) provides automatic camera calibration and can stitch video in real time in an equirectangular format. It is an extremely powerful piece of software that allows you to create templates and adjust the control points as needed.
Figure 5. VideoStitch’s Vahana VR real-time video-stitching software. Click the image to see it at full size.
With Vahana VR you can preview both the equirectangular projection as well as the spherical projection with a VR headset, so there is less guesswork involved. Real-time stitching allows you to record a single stitched output instead of multiple ISOs from each camera, thus reducing the amount of storage needed on your hard drive. The single stitched output file can be modified in post because there is no need to sort, sync, and stitch ISOs with additional software or rendering.
Although Vahana VR has a lot of advanced features that make the stitching easier, the current state of the software is in need of some basic features such as undo, and bug fixes that can improve its stability. Fortunately, they just received a fresh infusion of venture capital and are addressing some of the missing features.
Encoding
Elemental Technologies recently just announced that its streaming encoders are now equipped to deliver full-frame-rate Ultra HD HEVC and 4K virtual reality streams. Following the company’s recent acquisition by Amazon, I expect to see more integration o Elemental into Amazon Web Services and more embedded support for VR.
CDN
Wowza Streaming Cloud is compatible with most VR streaming systems and is a pay-as-you-go video delivery platform. It is easy to set up and its flexibility makes it a top choice for live streaming.
Viewing Platforms
There are several viable methods of viewing live 360° video. You can view it directly on a web browser with either a custom Flash or HTML5 player, or through an app. The current preferred method is to view the live stream within an app on your mobile phone typically provided by a VR company or a hardware manufacturer.
You can send streams to any 360° mobile app that supports streaming as well as playback of 360° videos. Codeplex’s VR Player supports a wide range of formats and geometric projections as well as live streaming as an Android app. Homido, VRplayer, and VRlive.tv are iOS and Android apps available in the Apple App Store or Google Play. All of these apps play recorded 360° video and live streams as well.
You can stream directly to desktop browsers through a player such as KrPano, which is a Flash- and HTML5/WebGL-compatible custom player, supporting RTMP and HLS playback.
Conclusion
So, what’s the answer to the million dollar question, Is 360 VR Live Streaming Ready for Primetime? My answer is, not yet—but it will be very soon.
Although there have been great advances in VR over the last couple years, many issues are still being worked out. Over time, the production process will definitely become easier, as well as much more stable and standardized. From the content creator’s perspective, VR streaming is still a work in progress, but consumers will be ready to embrace live 360° viewing when it is available on easy-to-access mainstream platform such as YouTube.
When YouTube integrates live VR streaming to its site with mobile support (which it inevitably will), and begins to feature high-profile VR events such as the music awards, concerts, and fashion shows, this in effect will create more compelling reasons to view VR live streaming. Only then will VR streaming be ready for primetime and become part of our everyday lives.