Vol.4
RICOH THETA Developer Interview
4K spherical videos,
Ask the developer.

MatsumotoIn case of regular 4K videos, the entire screen is within the 4K resolutions (3840 pixel x 1920 pixel). In case of the 4K spherical (360-degree) video on the other hand, part of the 4K video is cropped and displayed so the resolutions are more important than regular 4K videos.

The importance of resolutions means; When a 2K video is used as a 2K spherical video, the actual view range will have resolutions as (960 pixel × 540 pixel) or so. When a 4K video is used as a 4K spherical video, the resolutions of the actual view range will be enhanced to Full HD resolutions (1920 pixel × 1080 pixel) or so.

I think regular 4K videos unexpectedly have somewhat overengineered aspects for general users to handle in terms of the fact that regular 4K videos' resolutions are too high. On the other hand, I believe 4K spherical videos enable users to enjoy the benefits of 4K videos to the fullest.

Ricoh Company, Ltd. Smart Vision Business Group,
Product Development Center,
Device Development Department
Makoto Shohara

Shohara The resolutions of previous THETA videos in particular has been 2K resolutions (1920 pixel × 960 pixel). Development from this resolution to 4K resolutions should be legitimate. In term of resolutions, when we released THETA S in 2015, we also received requests from users, who asked “why 4K resolutions are not supported?”.

In those days, however, considering the spread of playback environments, we judged that it was too early to support 4K resolutions. When using THETA, many users prefer to connect it to a smartphone to view images. However, smartphone itself didn't provide sufficient support for 4K-based playback.

MatsumotoSpecifications to play back 4K videos were left behind. It seems that they have been spreading lately.

MatsumotoThe specifications of both image sensors (imaging elements) and processors for taken videos and images have improved this time. Despite this improvement, the current processing speed barely enables the playback of 4K videos. For this reason, placing a higher emphasis on “improved image processing” may end up producing clumsy moves when a video is played back.

THETA V has realized 4K 29.97fps (fps means the number of frames recorded per second). We were attentive in balanced processing so that improved image quality is provided while the number of frames per second for smooth playback is realized.

Still images and videos differ in processing details performed inside the device. We have created THETA V so that processing at a high speed is possible solely for 4K videos. For previous models of THETA, we emphasized clear shooting of still images, and we hope users are satisfied with THETA as a video camera from now on.

ShoharaAs we achieved our original image processing mainly by GPU (Graphics Processing Unit), we considered the performance of GPU was essential. As it was required to process 30 frames of a large-size image with 4K resolutions in a second, we also needed to improve GPU programs. In addition, we considered various ways of enabling effective signal processing for the entire system while reviewing memory transfer, processes, and threads.

MatsumotoYes, I do. I know processing becomes even harder though (laughs).

ShoharaI also want to improve resolutions. I hope to realize 8K resolutions for videos.

MatsumotoWe were much excited. After all, videos compared with those with 4K resolutions were videos recorded using previous models. We were so impressed that we felt as if the image quality of a television using a Braun tube had suddenly changed to that of terrestrial digital broadcasting.

ShoharaAs 4K resolution means reproducing fine details, I felt as if still images were moving. The impression I had was that the reality of images is totally different.

Ricoh Company, Ltd. Smart Vision Business Group,
Product Development Center,
Device Development Department
Kazuhiro Matsumoto

MatsumotoIt was really more than what we imagined, and our colleagues only said “Amazing!” as their first impression. Since then we have continued tune-ups for image quality, and I currently feel we have created higher impact images.

ShoharaThis time we have provided support for 4K UVC (USB Video Class, a standard for Web cameras and those that connect to USB devices). Especially we have provided support for UVC 1.5 because video compression codecs (H.264) is used. While previous models were also compatible with UVC 1.5, support was provided for full HD at most, and providing support for 4K UVC was our first experience. In fact, after starting development, we found our development members were embarrassed. There were no devices compatible with 4K UVC 1.5 that we could refer to. There, we recognized that “THETA V might be the first device that enables 4K UVC 1.5 output” (laughs).

ShoharaAccordingly, miscellaneous problems also occurred. Although there were no problems when standards and software were for full HD or lower, non-applicable problems and malfunctions occurred when standards and software were higher than those for full HD. We needed to respond to this problem. I think it was April of this year (2017) that Windows® and Mac drivers started to support 4K UVC 1.5. As the development of THETA V was ahead of these trends, what happened was....

MatsumotoWhen UVC-based live streaming did not work well, we simply could not figure out whether our THETA was wrong or the Windows® computer receiving data was wrong (laughs).

ShoharaAfter all, we conducted verifications rigorously while developing drivers for computers until THETA V was released.

ShoharaIn future, I hope “we are the first to provide what can be done”.

A previous challenge for computers to handle spherical videos was operational load for connecting fish-eye images to make a 360-degree spherical image. THETA S and its previous models had a piece of software embedded in the driver to connect shot images within the computer. As 4K spherical videos are data-heavy, operational load can be enormous if only computers perform these processes. Therefore, we have changed the previous specifications so that the main unit of THETA V can process this job, reducing the computer's load.

MatsumotoNumerically, 4K is twice as large as 2K, but 4K video's data amount is four times as large as that of 2K video. If a computer needs to process this amount of data fully, its operational load can be considerable.

ShoharaHowever, it is true that requirements are reasonably higher if live streaming is actually performed using 4K 29.97fps. For instance, using a high-spec computer is needed. Even though it is true, we cannot advance further unless there is a device like THETA V. I assume that we could provide a material for new challenges.

MatsumotoWe repeated test shooting earnestly. We often pursued problems from the final images where we found something strange. As I have been in charge of image quality for a long period of time, I first obtain intuitive insights of problems, delving into them later.

There is no goal in a process for pursuing image quality. The more you are involved in the process, the more eager you will become. It is because levels of issues to be pointed out rise as levels of products from each site continue to improve through a number of steps.... I wish to commit to it endlessly unless there is a deadline, which is the release date of the product (laughs). Image quality is the issue I was particular about to the end.

ShoharaWe were lucky because Mr. Matsumoto was really careful about details. He engaged in image quality designing, always comparing THETA V's image quality with that of THETA S. He performed quality adjustment until the very end in order to improve THETA V's image quality even just a little. Without Mr. Matsumoto, we could not have created a product that surpasses THETA S.

MatsumotoIt is an exaggeration, isn't it? (laughs)

ShoharaStarting from what product we will produce by using what technology or what members we will have for the development team, I have been involved in THETA V from its launch. I have been engaged in a broad range of activities from something like planning to development and designing.

In this model, we have incorporated new features such as 4K video, 360° Spatial Audio, Remote Playback , and so on. To tell the truth, I have been proposing these ideas since we were developing the prototype of THETA's first model. Part of these ideas have been taken up this time. There is still a list of unrealized ideas I have been thinking about since that time.

MatsumotoMr. Shohara often says "THETA is not a camera". He says it in a good sense, of course.

ShoharaFrom the point of view of “creating a camera”, we will be bothered with “parts that are unlike those of conventional cameras”. We build a wall by ourselves and cannot afford to move forward. THETA is a product that has surpassed the existing preconceptions. We aim to create a product to provide new experiences for users. Based on an accumulation of those experiences, THETA V takes shape.

In the first place, we want THETA V to be used as a regular spherical camera in the future. In addition, in order to provide new experiences such as Remote Playback, we have found Android^™ is an ideal system.

MatsumotoFrom the eyes of THETA V developers, what Android^™ can do can be basically performed on THETA V, too. There were many THETA V development members who have not used Android^™, and they say "THETA V is a smartphone more than we imagined". Before creating the main unit, we used a large smartphone while considering software applications. We were running applications on it. Those applications we developed in that way have been installed on the current THETA V.

Although THETA V is a device not equipped with a display, there exists a familiar screen inside the device that cannot be seen from users. In the initial stages of development, we could watch YouTube from an application normally used on a smartphone.

ShoharaWith an enhanced sense of presence, we will have higher levels of pseudo experiences of being on the spot than those we have had before. In order to realize sound localization in spherical images for THETA V, we have adopted an Ambisonics-based spatial audio recording system. Thanks to this system, we have realized sound localization so that we can identify the location of a detected sound in direction. Many users see part of spherical images and the spatial region to be seen is limited. Meanwhile we hear sounds from spatial regions that are not seen. Moreover, as we look back to the direction a sound comes from, we can see the high-resolution image of the object emitting the sound, I assume that 360-degree Spatial Audio is a good match with 4K spherical videos.

MatsumotoCompared with THETA S's ISP (Image Signal Processor), the ISP of THETA V has been advanced so that there are benefits in terms of image quality. After realizing 4K resolutions for videos, we have had a modified sense of resolution to check how sharp a still image is. Referring to videos, thanks to full HD output mode that is available, the sense of resolution for the same full HD video mode has improved greatly when THETA V is compared with THETA S.

Although the pixel number of the sensor installed on THETA V is the same as that of the sensor installed on THETA S, the interface connecting the sensor and the entire system has improved to enable high-speed data transfer. This transfer speed realizes 4K resolution presentation.

These features have been realized by utilizing the performance of the main processor Snapdragon. Additionally, by utilizing the image processing expertise we have cultivated through the production of SLR cameras, we have improved THETA V's basic features as a camera.

MatsumotoYes, we have, and we have decided to work hard for it. We specify ISO3200 sensitivity as the upper limit for still images as we wish to maintain a certain level of image quality. Specifying ISO6400 sensitivity for videos is not a matter of image quality but a result of emphasizing benefits of “seeing what we have failed to see up to date”.

ShoharaEven if video images are recorded in a considerably dark place, we can see them as fairly normal video images.

MatsumotoIt does not pose a problem at all to record video images on the street at night where street lamps are scattered. Recording is reasonably possible in a bed room where only a miniature light bulb as large as a night-light bulb is lit. Of course, we cannot record very bright video images.

ShoharaRecorded video images in this way may be closer to natural-looking images.

ShoharaH.265 (HEVC) is a video compression standard, and it was created as a format to support 8K resolutions. Using this standard, we can apply about double the data compression ratio without reducing image quality, halving the data amount.

However, this standard has just started to spread among general users. There are only a few devices to play back contents compressed using H.265. But, we have made THETA V compatible with H.265, expecting that H.265 will come into mainstream use in the future. Arguably, if the data amount is halved, a storage capacity is not consumed. Using H.265 means significant benefits for users.

ShoharaBLE (Bluetooth® Low Energy) was installed. With this, location information obtained using the GPS capability of a smartphone can be automatically embedded in image data. For this, you can keep your smartphone in your pocket. As BLE can reduce power consumption, it is also possible to reduce battery power consumption.

Although location information has already been included in THETA S's image data when shooting was performed using a smartphone, we wanted to include location information in image data only by pressing a button on the main unit. If an image was taken in a place where we often visit, we will know where it was taken, but if it was taken in a place where we visited for the first time, this feature will help us.

MatsumotoWhat I am going to say is an opinion of a person who has been using a conventional camera for a long period of time. THETA's advantage includes “taking a photograph of an object with the photographer”. Unless you are intentionally in hiding when performing shooting, you are definitely taken with the object when using THETA in a normal manner.

I have taken lots of photographs privately so far as well. But, I only have group photographs where I was captured. In some cases, I was not captured in group photographs. This was because I was the photographer of those photographs (laughs). I feel glad to find my figure there in a photograph when I look at it later.

ShoharaTHETA's advantage is that it is easy to use. I hope users record their daily lives as much as possible. If you record your daily life, there must be something you will notice later. There should be something you notice, for instance, like “such a thing was happening” or “we used a piece of furniture like this in those days”. I think it is very meaningful that “objects other than those a photographer intended to shoot are recorded” beyond the photographer's intention.