A new version of Google Photos is out in the wild and it's sporting a brand new text selection feature for those with access to Google Lens. A teardown of the APK also reveals that Photos isn't too far from getting a powerful tool for removing obstructions and ghosting effects from images, and it may soon have a new 'themed movie' generator that does all of the hard work of picking out content for short personalized videos.

What's New

Unofficial Changelog: (the stuff we found)

  • Text selection in Google Lens

Text selection in Google Lens

Just as Google Lens is making its way into the hands of more people, it's already getting one of its first feature enhancements. The latest version of Photos enables a new Text Selection mode for images containing easily identifiable text.

All you have to do is hit the Lens icon and wait for it to finish examining the photo. Hit the button for Text Selection and you'll see areas in the image highlighted and all of the text will appear in the lower window. You can go even further by tapping individual highlighted areas to get just that text, and even tap each row of lines to get down to just that selection. From here, text can be copied, shared, or passed into a Google search.

This is an expansion of the existing capability to identify phone numbers, email addresses, and some similar fields.

Teardown

Disclaimer: Teardowns are based on evidence found inside of apks (Android's application package) and are necessarily speculative and usually based on incomplete information. It's possible that the guesses made here are totally and completely wrong. Even when predictions are correct, there is always a chance that plans could change or may be canceled entirely. Much like rumors, nothing is certain until it's officially announced and released.

The features discussed below are probably not live yet, or may only be live for a small percentage of users. Unless stated otherwise, don't expect to see these features if you install the apk.

Object and reflection removal

Back at Google I/O 2017, Sundar Pichai opened the show with the subject of machine learning and how important it had become to Google's product development. One of the teases included a shot of a girl playing softball, but it was heavily obstructed by a chainlink fence. Moments later, the fence was completely wiped away from the scene, leaving an image that looked as though it was taken with a perfectly clear perspective. I've cued up the video below.

There hasn't been much talk about this object removal feature since I/O, but it looks like the Photos team is finally gearing up to release it fairly soon.

A new screen has been created for "Processing Settings" that will give users a way to make adjustments before performing any special processing. Two types of obstructions are currently listed: Occlusions and Photo Reflections. Occlusions are any objects in the foreground that partially block the scene you're trying to capture, like the chainlink fence from the I/O demonstration. I'm not entirely certain what is meant by Photo Reflections, but it's almost certainly how they refer to images where the photographer and other objects are caught in the reflection off of glass or water, which leaves that ghost-like image floating on a surface that should be transparent.

strings
<string name="reprocess_photo">Reprocess Photo</string>

<string name="preference_processing_settings">Processing Settings</string>

<string name="preference_obstruction_type_title">Obstruction Type</string>
<array name="preference_obstruction_type_entries">
<item>@string/preference_obstruction_type_occlusion</item>
<item>@string/preference_obstruction_type_photo_reflection</item>
</array>
<array name="preference_obstruction_type_entryvalues">
<item>OCCLUSION</item>
<item>PHOTO_REFLECTION</item>
</array>
<string name="preference_obstruction_type_occlusion">Occlusion</string>
<string name="preference_obstruction_type_photo_reflection">Photo Reflection</string>

<string name="preference_use_gpu_title">Use GPU for faster processing</string>
<string name="preference_use_gpu_summary">Use GPU to accelerate some parts of the processing pipeline</string>

<string name="preference_uniform_frames_title">Uniform Frame Selection</string>
<string name="preference_uniform_frames_summary">Select frames uniformly in time (no spatial reasoning)</string>

<string name="preference_capture_settings">Capture Settings</string>

<string name="preference_enable_4K_title">Enable 4K Capture</string>
<string name="preference_enable_4K_summary">Enable UHD 4K video capture (if supported)</string>

<string name="preference_show_progress_title">Show Progress</string>
<string name="preference_show_progress_summary">Visualize progress with a paint-like UI</string>

<string name="preference_show_tracks_title">Show Tracks</string>
<string name="preference_show_tracks_summary">Debug the online processing by displaying tracked points</string>

from: /xml/preferences.xml
<?xml version="1.0" encoding="utf-8"?>
<PreferenceScreen xmlns:android="http://www.zambhalha.com/apk/res/android">
<PreferenceCategory android:title="@string/preference_processing_settings">
<ListPreference android:entries="@array/preference_obstruction_type_entries" android:title="@string/preference_obstruction_type_title" android:key="obstructionType" android:summary="%s" android:defaultValue="OCCLUSION" android:entryValues="@array/preference_obstruction_type_entryvalues" />
<CheckBoxPreference android:id="@id/use_gpu_checkbox" android:title="@string/preference_use_gpu_title" android:key="useGPU" android:summary="@string/preference_use_gpu_summary" android:defaultValue="true" />
<CheckBoxPreference android:id="@id/uniform_frames_checkbox" android:title="@string/preference_uniform_frames_title" android:key="uniformFrames" android:summary="@string/preference_uniform_frames_summary" android:defaultValue="true" />
</PreferenceCategory>
<PreferenceCategory android:title="@string/preference_capture_settings">
<CheckBoxPreference android:title="@string/preference_enable_4K_title" android:key="enable4K" android:summary="@string/preference_enable_4K_summary" android:defaultValue="true" />
<CheckBoxPreference android:title="@string/preference_show_progress_title" android:key="showProgress" android:summary="@string/preference_show_progress_summary" android:defaultValue="false" />
<CheckBoxPreference android:title="@string/preference_show_tracks_title" android:key="showTracks" android:summary="@string/preference_show_tracks_summary" android:defaultValue="false" />
</PreferenceCategory>
</PreferenceScreen>

There are some other takeaways from the processing settings besides just the type of corrections that can be made. A simple starting point is that users will be able to choose whether or not to use their phone's GPU to speed up processing.

But more interesting in a second option that determines if frames are examined in chronological order or if the algorithm is allowed to look ahead to subsequent frames to track changes. I think this is interesting because it specifies frames... as in video. All we had before was a demonstration of a still image, but this basically confirms Google will also make these features available to video, as well.

Three other options are in the list, including toggles to enable 4K capture, show a progress meter, and to show separate tracks. These look more like developer settings, so I doubt they will be making it to users.

While it doesn't really reveal anything new, there are also a few more strings related to the Photo Reflection feature, which is also currently referred to as 'Dereflection.' Aside from the permission request for camera and storage access, all of this looks like it will be used for development and debugging, not something users will see.

strings
<string name="app_name">Dereflection</string>

<string name="permission_rationale_message">Dereflection needs to access the camera and device storage to take pictures</string>
<string name="permission_rationale_title">Permission Required</string>

<string name="new_dataset">Capture</string>
<string name="share_dataset">Share Dataset</string>
<string name="capture_completion">Success. Processing data.</string>
<string name="capture_summary">Capture summary</string>
<string name="compressing_dataset">Compressing dataset...</string>

Themed movies

Dynamically generated videos have been going around the Google product sphere for years, so it's something of a regular ritual to see yet another variant cropping up in a teardown. Judging by a stack of new text, "themed movies" will become a new option soon.

While there's a fair amount of text, the only details you're going to learn from it are that it uses facial detection and pet grouping to generate the movies. There is also a manual mode for producing videos, but I see nothing to explain how that would be any different from the video creation feature we already have. I think the main point of themed movies is that they automatically pick people and pets from your library rather than asking you to make the selections.

strings
<string name="photos_create_movie_bottom_sheet_title_concept">Create themed movie</string>
<string name="photos_create_movie_bottom_sheet_title_manual">Create movie manually</string>

<string name="photos_create_movie_concept_confirmation_title">Creating movie</string>
<string name="photos_create_movie_concept_confirmation_subtitle">It will take a few minutes to create your movie. We will notify you when it is ready</string>
<string name="photos_create_movie_concept_confirmation_overflow">+%d</string>

<string name="photos_create_movie_concept_error_dialog_connection_error">Please make sure that your device is connected to the internet and try again</string>
<string name="photos_create_movie_concept_error_dialog_default">Please try again later</string>
<string name="photos_create_movie_concept_error_dialog_enable_creation">This feature requires creations to be enabled in settings</string>
<string name="photos_create_movie_concept_error_dialog_enable_face_clusering">This feature requires face grouping to be enabled in settings</string>
<string name="photos_create_movie_concept_error_dialog_enable_pet_clusering">This feature requires pet grouping to be enabled in settings</string>
<string name="photos_create_movie_concept_error_dialog_not_enough_clusering">You do not have enough face groups in your Google Photos account</string>
<string name="photos_create_movie_concept_error_settings_button">SETTINGS</string>

Download

The APK is signed by Google and upgrades your existing app. The cryptographic signature guarantees that the file is safe to install and was not tampered with in any way. Rather than wait for Google to push this download to your devices, which can take days, download and install it just like any other APK.

Version: 3.10.0.177210800

Google Photos
Google Photos
Developer: Google LLC
Price: Free