Windows 11’s Recall feature has been causing controversy recently, so much so that Microsoft has actually halted the feature in its tracks (for now) – but a new discovery won’t fan any of those particular flames. In fact, it could well prove useful for those who eventually take the plunge with the now-delayed AI-powered functionality.
As discovered in the new preview build 26236 for Windows 11 (in the Canary channel) by regular leaker @PhantomofEarth on X, the new addition to Recall – which is still hidden in testing – is a ‘Search the web’ option.
Windows Recall is getting a new “search the web” action for text detected by screenray in snapshots. (disabled by default, build 26236.) pic.twitter.com/NsGM2CGHCvJune 12, 2024
See more
To recap, Recall is an AI feature specifically designed for Copilot+ PCs which regularly takes screenshots of the activity on your PC, files them in a library, and makes this searchable via Microsoft’s Copilot AI in Windows.
The new ‘Search the web’ facility allows the user to right-click on any text detected in a screenshot taken by Recall, and it’ll fire up a search on that selected text (in the user’s default search engine, presumably – though we don’t get to see the feature in action).
The ‘Search the web’ option is present in Recall’s right-click menu (in a snapshot) alongside the ‘Copy’ and ‘Open with’ options.
New AI settings in Windows 11
X user @alex290292 commented on @PhantomofEarth’s post with another interesting observation that there are also new AI-related settings in this Windows 11 preview build.
New under “Privacy & Security” pic.twitter.com/pB6FA22DxoJune 12, 2024
See more
These are in the Settings app, under ‘Privacy & Security’ where there’s a ‘Generative AI’ panel that allows for the fine-tuning of which apps are allowed to use generative AI capabilities. Apparently, you’ll also be able to review the last seven days of activity to see which apps requested to use generative AI.
To be able to see all of this for yourself, you’ll have to install the preview build and use a Windows configuration tool (ViVeTool) to enable ‘hidden’ Windows 11 features – not something we’d recommend for anyone but a keen enthusiast who’s comfortable with tinkering around in test builds.
Microsoft came out hot with its Windows Copilot 365 and Copilot for Windows announcements last year, and presented Copilot as a general virtual assistant to help with your whole digital life. So far, we’re yet to see Copilot reach its potential, but it looks like we’re one step closer: Microsoft is reportedly gearing up to add a powerful new feature that will allow users to take a screenshot, submit it to Copilot, and ask Copilot to explain what’s in the screenshot.
As far as we know, an “add a screenshot” button is rolling out to the general public – meaning you may already be able to try it. If not, you should be able to very soon. This button should appear in the Copilot panel, prompting you to select a part of the screen, confirming that you’ve captured what you want to discuss with Copilot, and then upload it to the Copilot or Bing right-side panel. When I tried it, I did also need to submit some text to go along with it, such as a question or additional context.
Once the screenshot is uploaded, you can talk about it and ask about anything within it or relating to it with Bing Chat or Copilot on Windows 11.
As you can see in the screenshot, the new button sits next to the existing image upload button. You can try the new feature at Microsoft’s Copilot website, or over at Bing Chat.
Windows Latest has their own demonstration of this new feature, and I had a go of my own. First, I took a screenshot selection which showed a full description of video creation platform HeyGen's YouTube channel and asked it to tell me two things: whose channel it is and what it’s about.
Copilot returned this:
This is the YouTube channel of HeyGen. It is a next-gen video creation platform that turns text into professional spokesperson videos in minutes. They offer premium avatars speaking in multiple languages and professional video templates for various use cases including marketing, e-learning, and corporate communication. You can find more information about HeyGen on their website.
This is pretty accurate, and reminded me of one feature in particular that I really like about Bing Chat and Copilot – they readily and very visibly provide sources and websites that you can visit to double check the information.
Once you make a selection of your screen, you can make markings on it and draw on it. You can also add specific instructional visuals to help Copilot understand your query, and you can move your selection window around to a different part of the screen altogether.
According to Windows Latest, Bing Chat recently got a ChatGPT-4 boost granting it a new level of functionality and this is likely making its way into Copilot as well. Apparently access is currently only granted to select users, and this development enables Copilot to engage in conversations about emotions. Currently, there is a limited pool of users who can try this for themselves and access is seemingly given at random, and it will be available to all who access Windows Copilot and Bing Chat very shortly.
Microsoft charts a course ahead with Copilot
Microsoft has been pretty definitive in its messaging that Copilot is a big deal for the company, and will be a central feature in several products like Microsoft 365 and Windows, but not just those.
In a pretty major (yet not terribly surprising) development, Microsoft is planning to add an actual physical Copilot button into the hardware of newly manufactured products as early as 2024. Microsoft is doing this in its continuing effort to make computing, especially AI-powered computing, simpler and more seamless for users. This was detailed and confirmed in a recent Windows Experience Blog post written by Yusuf Mehdi, Executive Vice President and Consumer Chief Marketing Officer at Microsoft.
For the rest of us not ready to throw our older Windows devices out quite yet for this new button, you can bring up Windows Copilot with the shortcut Win+C (if you have updated your Windows 11 version to one that has Windows Copilot included).
According to Microsoft itself, the introduction of the Copilot key will be the most notable upgrade to the Windows keyboard in almost thirty years. It likens this future introduction to the addition of the Windows Start key, which is putting a lot of faith in Copilot itself so I imagine we’ll continue to see major developments to Copilot throughout this year. I think especially with Copilot’s development, Microsoft is one of the most exciting companies to watch this year.
Windows 11 often has incoming changes hidden away behind the scenes of the operating system, and another of these has just been spotted – and it’s a big one pertaining to AI.
Windows Central stumbled upon a tweet by regular leaker PhantomOfEarth on X, who has been digging around in Windows 11 preview build 26016 (in the Canary channel).
Hidden in Canary 26016: a settings page for AI components. Currently incomplete and just shows the contents of the system components page.vivetool /enable /id:47688503 pic.twitter.com/VfAZkaa367December 13, 2023
See more
PhantomOfEarth discovered a Settings page for AI Components which, as mentioned, is hidden, but can be turned on using a Windows configuration tool (ViVeTool).
This lists system components related to AI functionality, although it doesn’t do anything when enabled (unsurprisingly, it’s tucked away in the background for a reason – namely it doesn’t work yet).
The leaker also found strings related to the page, with one of those being: “View and remove AI components that are installed on Windows.”
So, it seems that this panel in Settings (under System) will allow you not only to view any AI-related system components, but also uninstall them if you wish.
Analysis: A necessary choice?
The AI components listed in the screenshot provided by PhantomOfEarth include Windows Security, the Microsoft Store, Phone Link and Xbox Game Bar – suggesting maybe that these will be furnished with AI extras at some point? Or they could just be placeholders, which is probably a more likely story – though we can certainly see the Microsoft Store, for example, getting augmented with AI (that suggests apps you might like based on the usage of your PC, or past downloads).
The latter brings up a point that may worry some Windows 11 users, namely privacy and exactly what AI might be doing in terms of profiling you, and building up a more in-depth picture of your likes, dislikes and so on, extrapolating from that. We should note at this point that this discussion is entirely theoretical, of course, but the general point is that some folks won’t want AI in their operating system – either for privacy reasons, or because they don’t trust it, perhaps.
It makes sense, then, that Microsoft will cater for those who want to remove AI abilities and provide these uninstallation options. Not that the presence of this Settings page in testing means anything yet – it could be scrapped in preview. Indeed, it isn’t even present in preview builds yet, it’s hidden in the background.
That brings us to another point – it’s very early work on this feature. The likelihood is that a wider swathe of AI functionality – and these options – won’t fully debut until next-gen Windows is released. (That’ll be next year, in theory, although we’re still not sure whether this will be Windows 12 – though whatever the case, big plans are afoot for AI, going by the latest rumors).
An interesting observation Windows Central makes here is that we don’t know how far AI uninstallation capabilities will reach – and whether that might include getting rid of Copilot? Yes, Copilot is in the cloud right now (so not on your PC anyway, or at least its ‘brain’ isn’t, only the interface), but Microsoft seemingly has plans to make the AI local – and if so, it’s possible that it could be made removable.
We doubt it, mind you, seeing as Copilot is such a central aspect of the OS – but at least some components relating to AI should be viable for uninstallation if this new finding is anything to go by.
Windows 11 has a new preview build which further improves Voice Access, an area Microsoft has been putting a lot of effort into of late.
Preview build 22635.2915 (KB5033456) has just been rolled out to the Beta channel, and one of the additions is the ability to make customized voice shortcuts.
Using this feature, you can specify a trigger phrase for the command, and then the command itself.
Microsoft gives an example of an ‘insert work address’ command which when given automatically pastes in the specified address of your workplace. Anytime you need that putting into a document you’re working on, you just say the command – which is quite the timesaver.
Language support for Voice Access has also been extended, and now the following are included (on top of the existing languages): French (France), French (Canada), German, Spanish (Spain) and Spanish (Mexico).
Finally for voice features, multiple monitors are now supported, meaning that when you summon a grid overlay – for directing mouse clicks to certain areas of the desktop – you can do so on any of the screens connected to your PC. (Before now, the grid overlay could only be used on the primary display).
You can switch your focus to another monitor simply by using a letter (A, B, C and so on) or its phonetic equivalent (Alpha, Bravo, etc).
Microsoft further notes that there’s a drag and drop feature to move files or shortcuts from one display to another.
Elsewhere in build 22635, screen casting in Windows 11 has been improved, with a help option now in the Cast flyout from Quick Settings. This can be clicked if you’re having trouble piping your desktop to another screen and want some troubleshooting advice.
Users are also getting the ability to rename their device for the Nearby Sharing feature to help identify it more easily.
For the full list of changes and fixes in this Beta build, peruse Microsoft’s blog post.
Analysis: Custom capers
This is some useful work with Voice Access, and those with multiple monitors who use the feature will of course be very pleased, no doubt. Voice shortcuts is a powerful addition into the mix for voice functionality, too, and with this, there are a good deal of options.
Not just pasting a section of text as we mention in the example above, but also tasks can be triggered such as opening specified URLs in a browser, or opening a file or folder. You can combine multiple actions too, along with functions like mouse clicks or key presses. This is a feature we’ve been wanting for some time, so it’s great to see it arrive.
It’s also worth noting that Windows Speech Recognition has been removed from Windows 11 in this build, and when you open that old app, you’ll now get a message informing you of its deprecation, and recommending the far superior Voice Access capability instead.
We’re hoping that in the future, Voice Access is going to become an even more central part of the Windows 11 interface, and it seems a great candidate to be driven forward with AI – and maybe incorporated into Copilot.
Meta CEO Mark Zuckerberg recently took to Instagram to preview a potential virtual keyboard feature for Quest headsets.
Posted on his official account, the short clip shows Zuckerberg and Meta CTO Andrew Bosworth typing away on a VR keyboard while wearing a Quest 2 headset. The device was able to accurately track their finger movements and display what they were writing on screen without requiring any extra peripherals. According to Zuckerberg, he was able to achieve 100 wpm (words per minute) while Bosworth hit 120 wpm. To put that into perspective, the average typing speed of an adult is 40 wpm so it does perform well.
If development bears fruit, it could solve a longstanding problem with virtual reality.
Typing in VR is a slow process. You’re forced to enter inputs one at a time since floating VR keyboards can't match the speed of a physical device. Sure, you can purchase one of the best physical keyboards out there to get the speed that you want. But then you’re forcing yourself to carry around an extra peripheral alongside the VR headset just to get the user experience you want. Things can get cumbersome.
A work in progress
There is still work to be done over at Meta’s Reality Labs research unit where this tech was developed.
News site UploadVR points out in their report the headset requires “fiducial markers” to work properly. Fiducial markers are those black and white squares you see in the Instagram video. They assist the hardware in calibrating itself so it knows where to place the virtual keyboard. The end goal here would be to one day not need those squares for help so the VR helmet can project the keeb on any flat-enough surface.
Personally, we worry about typing feel. This technology already exists with laser keyboards that can project the keys onto a flat surface. The problem with these projections is typing feels terrible because you’re just mashing your fingers into a table, and we fear Meta’s feature will essentially be the same thing. This may be fine for the occasional email, but we can’t imagine using a VR keyboard for an entire day’s work.
VR peripherals
It's important to mention Meta is holding a two-day Connect virtual event from September 27 to 28. It's been confirmed the Quest 3 headset will make its debut at Connect, and perhaps a beta test for the VR keyboard will be announced then. An official launch date seems unlikely. As stated earlier, there's still work to be done.
We’re also curious to know if the company will finally show off its wristband device at the event.
If you’re not aware, Meta has been working on a wristband gadget that can read the electrical signals in a person’s arm to register inputs. The latest trailer for this gadget shows it can be used for simple gestures like twitching your finger to control a video game avatar. However, back in 2021, an earlier prototype displayed the ability to function as a virtual keyboard by using the same electrical signals. It’s unknown at this time if Meta scrapped the wristband feature in favor of the headset keyboard or if it’s still in the works.
Windows 11 has just got a new preview version in the Dev channel which adds some smart features, principally the ability to view your phone photos right on the desktop.
As you may recall, a big project in testing right now for Windows 11 is the revamp of File Explorer, part of which is the addition of a photo gallery feature. What Microsoft has done here is to introduce a new button that lets users add their phone photos to the gallery.
Click the ‘Add Phone Photos’ button and you’ll be given a QR code to scan with your smartphone, then you can set up the device to pipe photos across and view those mobile snaps in Windows 11’s photo gallery.
There’s other work on File Explorer elsewhere in preview build 23471, including some bug fixes, and another feature introduction – the ability to pop tabs out of windows (or merge them back), as you can with a web browser.
Note that the latter feature is an incremental rollout, so not every Windows Insider will see it yet. Microsoft did announce, though, that every tester in the Dev channel will now have the photo gallery in File Explorer (which has been a phased rollout, too, but one that’s now finished).
For the full details of all the fixes and tinkering in preview build 23471, as ever you can immerse yourself in Microsoft’s lengthy blog post.
Analysis: Microsoft adds while Apple takes away
The further work on File Explorer is welcome, and particularly the functionality allowing you to view phone photos right there on the Windows 11 desktop with a minimum of hassle.
There’s some interesting timing here – though no doubt coincidental – as well, in that as Microsoft is introducing this new feature, Apple has just announced that its My Photo Stream offering is being shuttered. (It’s now effectively on the chopping block, and will be shut down at the end of July).
Now, that might be an old service, and one not used much anymore, but it’ll still be something of a blow when it finally shuts it doors for the diehard set of niche Mac owners who still use My Photo Stream to view their phone photos on the desktop.
Of course, as Apple points out, My Photo Stream has been long been superseded by iCloud Photos, and you should turn that on instead. But some folks haven’t set up iCloud Photos because they don’t want to (enjoying just seeing recent snaps automatically on their Mac, courtesy of Photo Stream, without having to sync an entire – possibly huge – photo library to iCloud).
There’s also the fact that iCloud Photos requires payment (beyond the 5GB that you get free, a rather measly allowance these days), and some aren’t happy about that, either – and the apparent financial motive for Apple to make this move.
At any rate, whatever your view of how redundant or pointless My Photo Stream is at this point – indeed, you may never even have heard of it – the fact remains that Apple is taking away one path for easily viewing your phone photos on the desktop (that at least some folks still use), while Microsoft is going in the opposite direction and adding one.
Uber is ending the mystery of your ride-sharing rating score. If three drivers gave you 2 stars, you’ll now be able to see it. You won't, however, see which drivers gave you which scores.
After more than a decade in business, whipping out a smartphone and summoning an Uber driver has become second nature for many people, as has the sometimes-awkward process of giving the driver a rating after the trip. However, Uber’s star rating system is a two-way street as drivers also rate passengers. Until now, though, riders could only see an average of their scores. In a company blog post on Wednesday, Uber announced that riders could view a breakdown of their star scores in the app.
“Now, we’re making it easier than ever to see exactly how your rating is calculated, and for the first time, we’re showing you the good (and the bad) ratings you received,” the company wrote.
Understanding how Uber drivers perceive and rate you could be either comforting, “Oh, they notice I’m always on time and in the right place,” or demoralizing, “I guess drivers don't like me eating my foot-long heroes, in their back seats.” At the very least, it might lead to a more egalitarian relationship between riders and drivers.
If you’re ready for this experience, here’s how to access your detailed ratings in the Uber app.
Start by heading to Settings in your Uber app. There, select Privacy and then the Privacy Center. Swipe over to the section titled “Would you like to see a summary of how you use Uber?” In addition to a breakdown of ratings drivers have given you, this menu shows general statistics, such as how many trips you’ve taken, how long you’ve been a member, and more. You can also view information on individual trips you’ve taken and how you paid for them.
The feature is available to all riders and is part of a transparency push by the ridesharing company. Though Uber keeps your data for the duration of the time you have an active account, the ratings you see in the app only reflect an average of the last 500 trips. If you’re looking at your score and wondering how you can bump your average rating, Uber has some tips, most of which are common sense:
Don’t leave trash in the car when you leave
Buckle up
Don’t make the driver wait for you
Don’t be a jerk
Don’t slam the door
Uber also outlined the cities where riders get the best and worst scores from their drivers. Seattle and Washington, DC were among the worst, but New York City ranked the lowest. At the other end of the spectrum, riders in Nashville, St Louis, and San Antonio earned better scores.
Wherever you live, this Uber passenger information could help you make a difference in your and your next Uber driver's shared experience.