Screenshots suck. But they’re about to get a lot smarter

Getty Images / WIRED

Screenshots are ubiquitous. From shopping references to memes and from saving cat photos to serious research, they’re a routine and important quality-of-life feature on computers and mobile devices everywhere.
Android 12 should finally get full-page website screenshots, if Google can iron the bugs out before release. Samsung and Apple have had that feature for a while, but their recent patents show an interest in capturing metadata, from live links to site-specific information like street addresses and song titles.

Advertisement

Tap Scroll Capture when you screenshot a Samsung Galaxy, and the phone’s screenshot tool will stitch together a long screenshot to capture an entire website. Browse your screenshots in the Gallery, and you’ll see a Go To Website shortcut, a feature introduced in 2017.
Since iOS 13 in 2019, when you take a screenshot in Safari, a thumbnail pops up allowing you to select Full Page and save the resulting long website capture as a PDF.
Despite the screenshot’s long history (see Timeline below) and the fact that Apple launched mobile screenshots way back in 2007 on the first iPhone, amazingly these features represent the current state of the art when it comes to mobile screen capture. LG, Huawei and others have their own takes on long web page capture, and third-party apps like LongShot can do it for the rest of us.
XDA Developers’ Mishaal Rahman spotted the new scrolling screenshot capture feature – currently very crash-prone – in Android 12 Developer Preview 1, an early iteration of the OS that should launch in autumn 2021.

Advertisement

But while Google plays catch-up, other developers are already looking at making screenshots smarter.
Where next?
Link saving is perhaps the most in-demand smart screenshot feature. Whether you’re a lawyer or academic doing research, or shopping for this season’s coolest lockdown fashions, preserving the links from the page you’re looking at for future reference is vital.
Full, copyable text capture is similarly important, and the usual solutions here are variations on saving to PDF, a long-established technique that’s seeing progressive refinements to more accurately represent the graphical layout of websites.
On desktop computers, specialist screen capture tools such as Fireshot Pro can do both. Mobile is still getting there, with Apple’s long screenshots accurately capturing text, while Samsung devices will link you back to the original page when you view them in the Gallery.

Advertisement

But new and increasingly sophisticated approaches to extracting and attaching relevant metadata to a screenshot are being developed, such as Qualcomm’s unexpectedly interesting ‘Associating a captured screenshot with application-specific metadata that defines a session state of an application contributing image data to the captured screenshot’, which describes a method to restrict capture to the active window and grab relevant metadata, such as a street address, a website URL, the length of a video, a direct link to a chat session, data about a music track being played, and any text deployed in the screenshot.

Qualcomm’s active metadata extraction patent (both images)

Apple was recently granted a patent on its ‘Device, method, and graphical user interface for editing screenshot images’, which also describes an approach to storing metadata, such as GPS locations, URLs, device state and image alt text, and provide a way of annotating and storing or sharing both the screenshot and relevant data about it.

The patent suggests that metadata such as URLs and application download links may be shared alongside screenshots. Accompanying diagrams show automatic link creation to accompany a screenshot of a web page sent in an email.
It also details an interface for editing, annotating and sharing screenshots on a touchscreen. Screenshot annotation is already a feature of Markup in iOS, introduced in 2016 with iOS 10, but this more recent patent’s descriptions of metadata handling add a potential new dimension to the future of screenshot sharing on Apple’s portable devices.

Samsung patent images shows URL selection for addition to a screenshot

Samsung’s abandoned ‘Method and device of tagging links included in a screenshot of webpage’ shows us that the company has been looking into creating screenshots with active links embedded in them, and a system to allow users to select which links to include and how they store that metadata.

An abandoned patent application appears to show what would become Microsoft Snip & Sketch

With so much prior art surrounding screenshots, many of these patents may prove to be unsustainable, but, like Samsung’s 2013 ‘Electronic device and method for using captured image in electronic device’ patent application, which appears to presage its Go to URL feature and Microsoft’s ‘Capturing website content through capture services’, which similarly seems to describe its Snip & Sketch app, released for Windows in 2019, even these abandoned intellectual property applications can show us the shape of screenshots to come.
Screen Capture: a timeline
1981: MS-DOS on the IBM PC 5150 sends text from the screen buffer to your printer when you press the Print Screen key
1985: Apple Macintosh System Software 2.0 adds graphical screenshots, with different key combinations to print them or save them to disk
1990: Windows 3.0 now saves the entire screen area to your clipboard as a bitmap image when you press print screen, and the current window when you press Alt-Print Screen
1990: third-party Window screenshot tool SnagIt lets you save screenshots to disk; it would later introduce screen-area marquee selection capture and long screenshots
2000: scrot is released and quickly becomes ubiquitous on UNIX and Linux systems
2002: Apple’s Mac OS X Public Beta launches with Grab screenshot tool inherited from NeXTStep
2002: the Windows Snipping Tool gets its first release for tablet PCs
2006: Windows Vista launches with marquee selection and annotation via the Windows Snipping Tool
2007: the iPhone launches with screenshot capabilities
2008: Android launches without screenshotting support. People root their phones to add it
2011: Valve adds an integrated screenshot tool to Steam, helping to boost the nascent artform of in-game photography
2011: Android finally gets integrated screenshotting in version 4 (Ice Cream Sandwich)
2012: Windows 8.0 finally allows screenshots to be automatically saved to disk
2014-2015: Samsung begins introducing long screenshots to its TouchWiz UI for Android
2016: iOS 10 introduces screen annotations in 2016 via its integrated Markup tool
2017: Firefox adds in-browser screenshotting, with long screenshot support
2017: Google does the same in Chrome, but only as a developer tool
2018: Microsoft introduces Snip & Sketch, which adds extra annotation tools and Universal Windows Platform integration
2018: macOS 10.14 introduces new Screenshot tool with additional menus and screen recording features
2019: iOS 13 adds long screenshot support to Safari
More great stories from WIRED
💸 The bitcoin elite are spending millions on collectable memes
🍇 Scientists are growing grapes in space to save Earth’s wine supply
🔑 Reduce your chances of being hacked – use one of the best password managers

Advertisement

🔊 Listen to The WIRED Podcast, the week in science, technology and culture, delivered every Friday
👉 Follow WIRED on Twitter, Instagram, Facebook and LinkedIn

Get WIRED Daily, your no-nonsense briefing on all the biggest stories in technology, business and science. In your inbox every weekday at 12pm UK time.

by entering your email address, you agree to our privacy policy

Thank You. You have successfully subscribed to our newsletter. You will hear from us shortly.
Sorry, you have entered an invalid email. Please refresh and try again.

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Leave a comment

Why You Need A Website

Now