I was thinking for quite a while about this as an idea, and while the prototypes aren’t exactly perfect I’m definitely feeling positive about the results.
It’s basically google maps with sound!
Imagine being able to look at a map, and zoom in somewhere in the world to not only see a snapshot of the location, but also explore the sonic space in glorious spatial audio. Sure to take off in the audio geek circles…
Honestly though, it doesn’t sound like much but it truly is a different way to look at a picture! Being able to look around a 360 photograph, and hear that moment in time along with it, is seriously cool (I think).
But it wasn’t exactly easy, as you are about to read.
All of the 360 photographs started as a set of 15 separate photographs all taken on my Huawei P20 Lite. Not the worst camera in the world, but the visual quality of the end result has definitely suffered. I originally planned to use a panoramic phone app, called P360, but the automatically generated 360 pic was terrible quality unless you committed to a £4/month subscription to unlock HD capture mode. Lucky for me there was another way.
When you have Adobe Creative Cloud, anything is possible. Photoshop has a nifty tool within File>Automate called Photomerge, that detects the order of the photographs you give it and develops a panorama to its best ability. After some trial and error (about an hour’s worth, photomerge doesn’t appreciate similar looking shots) and some post-stitch cropping, a high-quality panorama is born! Photoshop even allows you to trial the panorama as a VR/360 image under the 3D menu, which was super handy before making the move to Premiere Pro.
While the hard part is over, we still only have a picture and not a video that audio can be attached to. So, over to Premiere Pro! Premiere has some great VR video support, once you have made sure to modify your imported files to be read as a monoscopic 360 video. After that, it was just a matter of adding the photograph to the sequence (that needs to be adjusted to support 360 video in the sequence settings) and dragging the clip to the desired length of time. I also decided to add a text layer that reads Please wear headphones, which seemed like a good idea until the video was uploaded to youtube and now sits a couple of miles above your head. Finally, the video is exported to H.264 format as a VR video (this is important to encode spatial audio to).
H.264 was the required video format to encode audio to, but funnily enough it wasn’t suitable for the Facebook 360 Spatial Workstation. So now begins the process of converting our H.264 format to the FB360 friendly format of DNxHD. For this, I managed to use the 10 free conversions in the trial of a FFMPEG conversion tool. Once into this format, the Facebook 360 Video Player will give the green light.
It was my intention to record all of the 360 audio using a (very expensive) Soundfield microphone, but when it came to it both of the portable batteries were completely dead and showed no sign of life after 12 hours of charging. Shame! As a compromise I decided to use some ambisonic library content that I already had, and instead went out on location armed with a single Aston Stealth microphone to capture more specific sounds that I could spatialise later in the process. It was important any pre-spatial sounds were recorded in mono to avoid phase-issues in the Facebook 360 Spatial Workstation.
With all of the sounds sources or recorded, it was time to put the audio in the photographic scene. Paired with the Facebook 360 Video Player, the FB360 Plugins in Reaper allow quick, easy and convincing sound spatialisation once you know how to set it all up. All spatialised audio must be routed to a Control buss that has the FB360 Control insert, which links to the Video player and decodes spatialised sound to binaural for the benefit of the user. After that it’s just a matter of using the Spatialiser plugin on each sound source, which you can then attach to an area in the scene along with specifying the distance and height of the source. When you’re all done you can bounce in 4 or 8 channel audio (I used 4 channels as YouTube only supports 1st order ambisonics), just make sure to uncheck Decode Binaural in the Control plugin before you bounce!
Last bit! The last step is to encode the spatial audio into the video, and add the appropriate metadata that YouTube needs to present the video as a VR stream to viewers. This is done in Facebook’s Audio Encoder, and requires the 4 channel audio bounce from Reaper and the H.264 formatted video file from the Premiere Pro project. Then it’s done! Phew.
So there you have it. The whole process. It’s definitely worth exploring the Facebook 360 Spatial Workstation, as its simple to use and free! The audio does get a little bit mangled on occasion at certain frequencies (the birds in the woods snapshot is an example) but for the most part it’s a brilliant bit of software.
Hopefully I can get the quality up and register more locations with a functional 360 microphone too!