Microsoft just gave Windows Copilot a ChaGPT-4 boost and the ability to explain screenshots

Microsoft came out hot with its Windows Copilot 365 and Copilot for Windows announcements last year, and presented Copilot as a general virtual assistant to help with your whole digital life. So far, we’re yet to see Copilot reach its potential, but it looks like we’re one step closer: Microsoft is reportedly gearing up to add a powerful new feature that will allow users to take a screenshot, submit it to Copilot, and ask Copilot to explain what’s in the screenshot. 

As far as we know, an “add a screenshot” button is rolling out to the general public – meaning you may already be able to try it. If not, you should be able to very soon. This button should appear in the Copilot panel, prompting you to select a part of the screen, confirming that you’ve captured what you want to discuss with Copilot, and then upload it to the Copilot or Bing right-side panel. When I tried it, I did also need to submit some text to go along with it, such as a question or additional context. 

A screenshot in a Microsoft Edge window with a Copilot panel open on the right, with arrow pointing to new

(Image credit: Future)

Once the screenshot is uploaded, you can talk about it and ask about anything within it or relating to it with Bing Chat or Copilot on Windows 11

As you can see in the screenshot, the new button sits next to the existing image upload button. You can  try the new feature at Microsoft’s Copilot website, or over at Bing Chat

Windows Latest has their own demonstration of this new feature, and I had a go of my own. First, I took a screenshot selection which showed a full description of video creation platform HeyGen's YouTube channel and asked it to tell me two things: whose channel it is and what it’s about.

Copilot returned this: 

This is the YouTube channel of HeyGen. It is a next-gen video creation platform that turns text into professional spokesperson videos in minutes. They offer premium avatars speaking in multiple languages and professional video templates for various use cases including marketing, e-learning, and corporate communication. You can find more information about HeyGen on their website.

This is pretty accurate, and reminded me of one feature in particular that I really like about Bing Chat and Copilot – they readily and very visibly provide sources and websites that you can visit to double check the information. 

Once you make a selection of your screen, you can make markings on it and draw on it. You can also add specific instructional visuals to help Copilot understand your query, and you can move your selection window around to a different part of the screen altogether. 

According to Windows Latest, Bing Chat recently got a ChatGPT-4 boost granting it a new level of functionality and this is likely making its way into Copilot as well. Apparently access is currently only granted to select users, and this development enables Copilot to engage in conversations about emotions. Currently, there is a limited pool of users who can try this for themselves and access is seemingly given at random, and it will be available to all who access Windows Copilot and Bing Chat very shortly.

Microsoft Bing logo on a white smartphone screen

(Image credit: Shutterstock / Primakov)

Microsoft charts a course ahead with Copilot

Microsoft has been pretty definitive in its messaging that Copilot is a big deal for the company, and will be a central feature in several products like Microsoft 365 and Windows, but not just those. 

In a pretty major (yet not terribly surprising) development, Microsoft is planning to add an actual physical Copilot button into the hardware of newly manufactured products as early as 2024. Microsoft is doing this in its continuing effort to make computing, especially AI-powered computing, simpler and more seamless for users. This was detailed and confirmed in a recent Windows Experience Blog post written by Yusuf Mehdi, Executive Vice President and Consumer Chief Marketing Officer at Microsoft.

For the rest of us not ready to throw our older Windows devices out quite yet for this new button, you can bring up Windows Copilot with the shortcut Win+C (if you have updated your Windows 11 version to one that has Windows Copilot included). 

According to Microsoft itself, the introduction of the Copilot key will be the most notable upgrade to the Windows keyboard in almost thirty years. It likens this future introduction to the addition of the Windows Start key, which is putting a lot of faith in Copilot itself so I imagine we’ll continue to see major developments to Copilot throughout this year. I think especially with Copilot’s development, Microsoft is one of the most exciting companies to watch this year. 

YOU MIGHT ALSO LIKE

TechRadar – All the latest technology news

Read More

Duolingo’s new GPT-4 AI will happily explain why your Spanish is wrong

Duolingo is launching a new virtual tutor that aims to replicate real-world scenarios to help students learn better. And it’s all powered by the recently released GPT-4 AI model.

Making its home in the new Duolingo Max subscription tier, the tutor consists of two features: Explain My Answer and Roleplay. The former, as its name suggests, gives users the opportunity to, if they’re confused by something in DuoLingo's initial response, ask the chatbot Duo to give a detailed explanation of why their answer was right or wrong. In an example video, the AI explains why select Spanish verbs must be conjugated a certain way given the context of the sentence. 

Duo, however, is not universally available on all language exercises, just certain ones. On those few, a Explain My Answer button will appear at the bottom of the screen after you attempt an exercise.

Roleplay, on the other hand, allows users to engage in a realistic conversation with the AI so they can practice their language skills. According to the post, no two chats will be exactly the same. In one instance, you could be talking to “waiter” as you order coffee at a French café or discussing vacation plans in Spanish with a “friend.” And at the end of every Roleplay, Duo will give you some feedback based “on the accuracy and complexity of [your] responses, as well as tips for future conversations.” 

Limited release

Do be aware that the GPT-4 AI behind Duo is not perfect. For the new release, research laboratory OpenAI took the time to improve GPT-3’s chat abilities so it can produce more natural-sounding text, similar to how people normally speak – at least in English. GPT-4 can create language guides such as utilizing English mnemonics for Spanish words. However, as seen on Twitter, those mnemonic guides can be pretty hilarious and not always in a good way. Duolingo admits its virtual tutor will make some mistakes. As such, the company is asking users to give the AI some feedback which you can do by selecting either the “thumbs-up” or “thumbs-down” emoji at the end of every Explain My Answer session.

Currently, Duolingo Max is seeing a limited roll-out. The AI will only be available in either Spanish or French for English speakers on iOS, but there are plans to “expand to more courses, language interfaces, and platforms in the coming months”, according to a company representative.

To subscribe to the tier, you have two methods. You can either pay $ 29.99 for Duolingo Max or $ 167.99 for the whole year. Breaking everything down, the yearly cost comes down to $ 13.99 a month. Additionally, you also get every feature under Super Duolingo which includes “unlimited hearts [for lessons], no ads, and [a] personalized review through the Practice Hub.”

While we have you, be sure to check out TechRadar’s list of the best AI writers for 2023 if you need content done fast. 

TechRadar – All the latest technology news

Read More