Artificial intelligence (AI) is literally everywhere nowadays. What began as research into cutting-edge science and computing way back in the 1950s, has slowly but surely crept into our lives today. Over the past few years, AI has picked up pace, and almost every field has started imbibing the nuances of AI at varying levels of maturity.
The conversation around AI, and artificially-intelligent algorithms improving and enhancing mobile photography has been around for a while. Among the very first to use AI as a mainstream catchphrase were the likes of OPPO and Vivo, who claimed to have used AI algorithms in their front-facing cameras to improve the quality of selfies. That, however, was only a rudimentary form of application of AI, and today, ‘AI camera’ is finding common parlance in actually taking smartphone photography forward.
The existing AI technology
At present, flaunting the use of AI in a camera has become one of the foremost keywords for any new smartphone aspiring to grab the attention of customers. Smartphone cameras themselves have evolved leaps and bounds, to a point where we are now well past the novelty of dual-camera setups. Speaking in terms of hardware, we have the likes of the Huawei P20 Pro (review) that has three, individual rear-facing cameras working in tandem, while the very recently-announced Samsung Galaxy A9 (2018) (first impressions) does one better to include four cameras. Four.
Further development on the smartphone imaging hardware front have included separating the image processing unit from the main processor (like Google’s Visual Core used in its Pixel line of devices), and Sony’s patented, stacked CMOS (complementary metal oxide semiconductor) sensors that use a high-bandwidth DRAM in between the image sensor’s layers to increase buffer memory, thereby allowing a camera unit to store more information per unit time, and in turn, produce better results.
However, research and development centres have certainly observed that there is a limit to which one can push the boundaries of hardware, and while having a higher number of individual camera units do account for versatility, it does not necessarily mean excellent photography performance. Hence steps in software, and the myriad optimisations and customisations that it can offer. At the centre of this software development, is the AI-powered camera. Let’s take a closer look at how different brands are using this in their offerings.
One of the applications of the AI camera can be found in the LG G7 ThinQ (review). In this device, LG’s AI Pack 2.0 includes algorithms that the company claims has been trained with machine learning through 100 million images, to possess over 1,000 image identifiers divided into eight different scenes. For those unaware, ML is a subset of artificial intelligence which learns from past cases to automatically understand what to do if something happens again in the future. In layman terms, when you open the camera app of the G7 ThinQ, the app’s algorithms will scan the frame, identify the proper settings from the thousands of scene identifiers that it has been trained with, and apply the optimum parameters of saturation, contrast, sensitivity and colour. Hence, you get a camera that is intelligent — one that can think for itself and produce the perfectly balanced shot (at least on paper).
ASUS too, has a similar implementation of its AI camera, which also takes into account its user preferences and applies that to its database of preset scene parameters to produce an optimised photograph. This means that the user will not need to worry about manually adjusting the camera settings every time a photograph is to be taken, and the AI algorithms take care of it, all within a split second. Huawei has a similar implementation of AI with its Master AI 2.0 image processing system, which also includes the AI’s image recognition capabilities of segmenting elements within a frame, hence recognising objects and allowing the camera to optimise photographs better.
Further down the line are more sophisticated systems like what Samsung implemented in the Galaxy S9 duo (review), which are already equipped with Dual Pixel sensors for enhanced focusing speed and low light performance. Samsung paid keen attention to image noise and low-light photography – factors that are far more controllable in dedicated cameras. The AI algorithms implemented by Samsung combine the software with a powerful image signal processor (ISP) to bring forth a multi-shot mechanism, wherein low light, the camera actually takes 12 individual shots of the same subject (and frame), and the AI algorithms then fuse all the data together to produce the final photograph with the least-possible image noise, while preserving brightness, colour and details. This is one key example of AI solving one of smartphone photography’s engineering barriers, which are imposed by the small size of mobile imaging sensors (typically 1/2.3″-type).
Conversationally speaking, AI and ML have been steadily implemented by smartphone makers to simultaneously improve the quality of phone photography, and make the niche art of photography easier to understand and more accessible to the masses. With such technologies at hand, one does not need an interchangeable lens camera (ILC) to shoot masterful images – simply whip out a smartphone, fire up the camera app, and at the press of the shutter button, an AI-optimised photograph is produced.
Now, while the primary objective of AI cameras has been to compensate for human flaws and make the overall process of photography easier, two companies, in particular, have highlighted the use of AI in ‘computational photography’, which truly show the potential that AI brings to the table.
Beginning with Apple’s efforts, the new iPhone XS (review) takes a unique route towards handling colour tone and details. It fuses the best of every world – a high-buffer sensor, a higher-precision lens and sophisticated AI-powered algorithm to give rise to this. Here, when the camera app is open, the buffer captures four frames the moment you press the shutter — basically, it is a constant, rolling shutter that captures the photograph at literally zero lag. This helps the camera determine the best photograph from the captured array, which helps do away with annoying bits like blurred hands and closed eyes in a photograph.
Instead of simply throwing away the rest of the shots, Apple uses them for computational analysis of the frame. It uses contrast-based segmentation of elements in the frame, followed by AI-powered object recognition. This helps the camera segregate mundane elements and breathing space (sky and other empty bodies) from coloured surfaces and skin tones. This is where Apple makes the strongest use of computational analysis — by using its proficient AI algorithms to adjudge the correct skin tone from available colour and brightness data of the entire frame. It then applies multiple exposure settings to the segregated elements of a frame to find the perfect balance of light, colour tone and sharpness.
What a user will see from this is exemplary colour accuracy, reduced image noise (since the merging of multiple-exposure shots lead to noise reduction as well) and a very well-balanced overall photograph, without oversaturating any colour tone. All of these operations are done without any visible delay, aided by the added sensor buffer and the new ISP.
Google’s new Pixel 3 (review) has a similar mode of operation, too, but presents an even more advanced scene-recognition algorithm that identifies fleeting expressions and smiles to capture instantaneous moments with Top Shot. Even more remarkable is Night Sight — a low-light photography mode that uses multiple long-exposure shots to literally show you more in darkness, than what the human eye can see. There is a loss in details and higher sensor noise in the Night Sight photographs, but that is inevitable in sensors this small. Yet, the use of AI in the Pixel 3 camera turns these photos into usable ones as well, with accurate colour balance and significantly higher details than before.
Another interesting factor to note is the use of only a single camera unit in the Pixel devices. While most devices have shifted to using multiple camera units in smartphones, Google has deemed it unnecessary, and instead focused on developing the software to produce similar, or even better results. For instance, Google has trained its imaging algorithms using ML to process or gauge depth in the photographs that the Pixel smartphones capture, and this perceived depth is then used to process blurred backgrounds or bokeh, in the ever-popular portrait modes.
Another aspect where Google has been working on since 2016 is the super resolution mode, or using ML for upsampling of low-resolution images. Its efforts with ‘Raisr’ upsampling algorithms has matured into ‘super res zoom’ in the Pixel 3 smartphones. With this, Google has attempted to produce better resolution digital zoom, without using a second telephoto lens. In a typical digital zoom, a digital crop is placed on a small area of a frame and then expanded in size to give the effect of a zoomed-in image. With Google’s ML-based upscaling technology, the algorithms use reference pixels of the original, high-resolution image to produce crisper, sharper upscaled pixels. This, in turn, produces a much sharper digitally-zoomed photograph, which is done entirely with the expertise of Google’s ML algorithms.
All this work around the AI camera os a boon for the masses, and even avid photographers. With AI ironing out human flaws and even improving on mechanical aspects by fusing multiple shots, varying exposures and identifying precise colour tones, smartphone photography stands at a balance where it truly rivals the photography performance of an average, entry-segment camera.
What it truly means is that the average user can now take stellar photographs without ever having to bother about the correct shutter speed, light flicker (the new Pixel devices even come with an anti-flicker sensor), white balance and exposure. The ‘Auto’ mode is the most superior one in cameras right now, powered by artificially-intelligent algorithms that are far more proficient at identifying flaws than humans.
Would this, though, mean the end of dedicated cameras? Well, there’s still no replacing full-frame and medium-format sensors for ultra-high resolution shots, but on a mainstream (and even hobbyist) note, the progress of AI cameras in smartphones has truly taken mobile photography to an all-new level.