Infrequently Noted

At first glance, the market for mobile browsers looks roughly functional. The 85% global-share OS (Android) has historically facilitated browser choice and diversity in browser engines. Engine diversity is essential, as it is the mechanism that causes competition to deliver better performance, capability, privacy, security, and user controls. More on that when we get to iOS.

Tech pundits and policymakers form expectations of browsers on the desktop and think about mobile browser competition the same way. To recap:

• Users can freely choose desktop browsers with differing features, search engines, privacy features, security properties, and underlying engines.
• Browsers update quickly, either through integrated auto-update mechanisms or via fast OS updates (e.g., ChromeOS).
• Browsers bundled with desktop OSes represent the minority of browser usage, indicating a healthy market for replacements.
• Popular native apps usually open links in users' chosen browsers and don't undermine the default behaviour of link clicks.^[1]

Each point highlights a different aspect of ecosystem health. Together, these properties show how functioning markets work: clear and meaningful user choice creates competitive pressure that improves products over time. Users select higher quality products in the dimensions they care about most, driving quality and progress.

The mobile ecosystem appears to retain these properties, but the resemblance is only skin deep. Understanding how mobile OSes undermine browser choice requires a nuanced understanding of OS and browser technology. It's no wonder that few commenters are connecting the dots.^[2]

How bad is the situation? It may surprise you to learn that until late last year only Safari could be default browser on iOS. It may further disorient you to know that competing vendors are still prevented from delivering their own engines on iOS. Meanwhile, on Android, the #2 and #3 sources of web traffic do not respect browser choice. Users can have any browser with any engine they like, but it's unlikely to be used. The Play Store is little more than a Potemkin Village of browser choice; a vibrant facade to hide the rot.

Registering to handle link taps is only half the battle. For a browser to serve as the user's agent, it must also receive navigations. Google's Search App and Facebook's various apps for Android undermine these choices in slightly different ways.^[3] This reduces the effectiveness of privacy and security choices users entrust in their browsers. Developers also suffer higher costs and reduced opportunities to escape Google, Facebook, and Apple's walled gardens.

Web engineers frequently refer to browsers as "User Agents", a nod to their unique role as interpreters of developer intent that give users the final say over how the web is experienced. A silent erosion in the effectiveness of browser choice has transferred this power away from users, re-depositing it with dominant platforms and app publishers. To understand how this sell-out happened (quite literally) under our noses, we must look closely at how mobile and desktop differ.

The Baseline Scenario #

The predominant desktop situation is relatively straightforward:

Browsers handle links, and non-browsers defer loading http and https URLs to the system, which in turn invokes the default browser. This flow is the central transaction that gives links power and utility. If any of the players involved (OSes, browsers, or referring apps) violate aspects of the contract, user choice in browsers becomes less effective.

"What, then, is a 'browser'?" you might ask? I've got a long blog post brewing on this, but jumping to the end, an operable definition is:

A browser is an application that can register with an OS to handle http and https navigations by default.

On Android this is expressed via specific intent filters in the manifest and listing in the BROWSABLE category. iOS gained browser support in late 2020 (a dozen years late) via an Entitlement.^[4] Windows and other Desktop OSes have similar (if less tidy) mechanisms.

No matter how an OS technically facilitates user choice, it's this ability to choose that defines browsers as a class. How often links lead users to their preferred browser controls the meaningfulness of this choice.

Android's "In-App Browser" Problem(s) #

The history of mobile computing starts from an incredibly resource-constrained point. First-generation iOS and Android smartphones were slow single-core, memory-impoverished affairs, leading mobile OSes to learn new tricks to facilitate responsive computing. Android and iOS adopted heuristics to kill and reclaim RAM used by non-foreground apps when resource pressure intensified.

This background task killing behaviour created unique problems for link-heavy apps. Launching the user's browser placed linking apps in the background, increasing friction in returning to the sending app, as browser UI did not provide affordances for returning to referring applications. Being put in the background also increases the likelihood of being killed.^[5] Returning to the source app while in this state can feel excruciating. It can take seconds to re-start the original app and restore the UI state, an experience that gets worse on low-end devices that are most likely to evict apps in the first place.

Engagement-thirsty apps began including "In-App Browsers" ("IABs") to address these challenges. Contrary to any plain-language understanding of "a browser", these IABs cannot generally be installed as the default handler for links, even when OSes support browser choice. Instead, they load content referred by their hosting native app in system-provided WebViews.

The benefits to apps that adopt WebView-based IABs are numerous:

• WebViews are system components designed for use within other apps. They do not place embedders in the background where the system may kill them to reclaim resources. This reduces friction and commensurately increases "engagement" metrics.^[6]
• As they are now "the browser", they can provide UI that makes returning to the host application easier than continuing on the web.
• Because they lean on the system-provided WebView component, they do not need to pay the expense of a heavier app download to support rendering HTML, running JavaScript, decoding images, or loading network resources.
• Apps can customise UI to add deeper integrations, e.g., "pinning" images from a hosted page to Pinterest.
• On Android today and early iOS versions, WebViews allow embedders to observe and modify all network traffic (regardless of encryption). Apps can also monitor user input, resulting DOM, and system auto-filled credentials.

To the extent that users are comfortable with apps not remembering their previously-stored passwords, login state, privacy preferences, extensions, or accessibility configurations, this can be a win-win.

Conversely, the web feels broken when any one of those conditions is not met.

Thanks to the light (bordering on non-existent) attribution back to the hosting app, along with disjoint and buried user controls to disable this misfeature, users may think the web isn't worth visiting, or that their browser broke.^[7]

1...2...3...Party Time! #

WebViews are the source of much confusion in debates around apps and browser choice. Thankfully, the situation is only complicated rather than complex.

There are two dimensions in play:

• Can the app register with an OS to handle http/https navigations by default?
  • If so, it's a browser regardless of the underlying engine.
  • If not, it's something else; a "content renderer" or an IAB.
• Does the app include its own web engine?
  • If so, it's integrated — e.g., an "Integrated Browser".
  • If not, it's WebView-powered, e.g. a "WebView Browser" or "WebView IAB".

So, a browser can be WebView-based, and so can an IAB. But neither has to be.

What is a WebView? To quote the Android documentation, a WebView is...:

A View that displays web pages. ... In most cases, we recommend using a standard web browser, like Chrome, to deliver content to the user.

WebViews have a long history in mobile OSes, filling several roles:

• Rendering HTML on behalf of the first-party application developer.
• Displaying cooperating, second-party content like ads.
• Providing the core of browsers, whose job is to display third-party content. The original Android Browser used early-Android's system WebView, for instance.

The power dynamics of these situations are starkly different, even though "web technology" is used to render content in each case.

The use of a "raw" WebView is entirely appropriate for first and second-party content. Here, native apps are doing work related to their core function; storage and tracking of user data are squarely within the four corners of the app's natural responsibilities. Furthermore, the content developer is unlikely to find limits presented by the WebView to be unwelcome or unreasonably immutable (via collaboration with the app developer). Instead of breaking content, WebViews are likely to facilitate it in these scenarios faithfully.

All bets are off regarding WebViews and third-party content, however. To understand why it helps to know that WebViews are not browsers.

WebViews contain core browser features, along with hooks that allow embedders to "light up" many more. However, producing a complete and competitive WebView-based browser requires additional UI and glue code. In particular, features that require permission-gated access to privileged services need explicit support from embedders to work as specified.

Features in this category include:

• Privacy and site quality support, including "acceptable ads" enforcement and browser history clearing.
• Security indicators (TLS padlock, interstitial warnings, etc.)
• Basic navigation and window management features to (e.g. window.open() and <a target="_blank"> which are critical to some site monetization features).
• Friction reducing OS integrations such as:
  • Web Payments (streamlined e-commerce checkout)
  • Web OTP (for easier/faster sign-in)
  • Web Share
• Hardware access APIs, notably:
  • Geolocation
  • Camera/mic (getUserMedia())
  • Web Bluetooth
  • WebUSB
  • Web Serial
  • WebHID
  • WebMIDI
  • Web NFC
  • Filesystem Access
• Re-engagement features including:
  • PWA installation and home screen shortcuts for sites
  • Push Notifications

Few (if any) WebView browsers implement all of these features, even when their underlying WebViews support bindings for them.

The situation is even more acute in WebView IABs, which tend not to fully support features from these categories even when they appear available to developers via script. Worse, debugging from these IABs is challenging, compounded by a lack of awareness about how much traffic may come from these sources.

How can that be? Web developers are accustomed to real browsers in the desktop mould. Standard tools, analytics packages, and feature availability dashboards do not make mention of IABs, and the largest WebView IAB promulgators (Facebook, Pinterest, Snap, etc.) have invested almost nothing in clarifying the situation.

It's vital to understand that neither users nor developers chose Facebook, Pinterest, or Google Go as a browser. The flow that WebView IABs present denies users agency over their choices, and technical limits imposed by them often prevent developers from opening content in real browsers.

No documentation is available for third-party web developers from any of the largest WebView IAB (ab)users. This absence mirrors the scandalous free-riding of these app publishers regarding browser feature support, which is perhaps not surprising. It is, however, all the more egregious for the subtlety and scale of breakage.

If Facebook, the third largest "browser"-maker for Android, employs a single developer relations engineer or doc writer to cover these issues, I'm unaware of it. Meanwhile, forums are full of melancholy posts recounting myriad ways these submarine renderers break features that work in other browsers.

Having been given "the first 80%" of a browser, with development and distribution of critical components subsidised by OS vendors, WebView IABs near-universally fail to keep up their end of the bargain with either users or developers. First-party webdevs can collaborate with their app-development colleagues to build custom access for any exotic feature supported by the OS. Second-party developers expect less (ads are generally not given broad feature access). But third-party developers? They are as helpless as users are to understand why an otherwise browser-presenting environment appears subtly, yet profoundly, broken.

Maximiliano Firtman @firt

There are still users browsing with a Chrome 37 engine (7 years ago), not because they don't update their browsers but because it's Facebook Mobile Browser on Android 5 using a webview. Facebook does NOT honor user browser choice leaving that user with an old engine. +

52 12:12 PM · Jul 15, 2021

These same app publishers request (and heavily use) features within real browsers they do not enable for others, even when spotted the bulk of the work. Perhaps browser and platform vendors should consider denying these apps access to capabilities they undermine for others.

WebView IAB User Harms #

The consequences of WebView IABs on developers are noteworthy, but it's the impacts on users that inspire confusion and rage.

Consider again the desktop reference scenario:

Clicking links from apps transfers control to an external browser which dutifully applies the user's stored preferences and accumulated state. Login credentials for example.com are not forgotten when a link is followed from an email. The same unified experience ensures that saved addresses and payment information are readily available. Most importantly, accessibility settings and privacy preferences are consistently applied.

By contrast, WebView IABs fracture user state, storing it in silos within each hosting application, creating a continuous partial amnesia.

The confusion that reliably results is the consequence of an inversion of the power relationship between apps and websites.

Does any user expect that everything one does on any website loaded from a link in the Facebook app, Instagram, or Google Go can be fully monitored by those apps? That all passwords shared and the full scope of sites visited from the first page can potentially be recorded and tracked?^[8] To be clear, there's no record of these apps using this extraordinary access in overtly hostile ways, but even the unintended side-effects reduce user control over data and security.

Retaining onward links is not objectionable in programs that also offer themselves as browsers, but the WebView IAB sleight of hand is to act as a browser when users least expect it, but never to cop to the power and privacy implications of the responsibilities browsers accept.

CCT: A New Hope? #

Libraries emerged to facilitate the construction of WebView IABs, leading to a recognition by OS and browser vendors that users were becoming confused and developers irate.

To address this challenge, Apple introduced SFSafariViewController and Google followed suit with the inartfully-named Chrome Custom Tabs protocol and helper library. Both systems allow native app developers to skip the drudge work of building a WebView IAB system and instead work with the OS to invoke the user's default browser to load web pages within the context of the host app. Similarly to WebView IABs, CCT and SFSVC address background eviction and lost app state. However, because they invoke the user's actual browser, they also prevent user confusion whilst delivering the complete set of features supported by proper browsers.

These solutions come at the cost of some flexibility for app developers who lose access to read network traffic between users and third-party sites. They also cannot inspect and change page content trivially, removing the ability to add new, non-standard features to their IABs. Counterbalancing these concerns, CCT and SFSVC restore user choice^[9] and ensure developers access to the complete set of browser features.

Et Tu, Google? #

CCT sounds pretty great, huh?

Well, it is. At least in the default configuration. Despite the clunky inclusion of "Chrome" in the name, the CCT library and protocol are browser-agnostic. A well-behaved CCT-invoking-app (e.g., Twitter for Android) will open URLs in the CCT-provided IAB-alike UI via Firefox, Brave, Samsung Internet, Edge, or Chrome if they are the system default browser.

That is unless the native app overrides the default behaviour and invokes a specific browser.

Ada Rose Cannon @AdaRoseCannon

@slightlylate I recently was talking to my Dad about the Web and asked what browser he uses and he showed me what he does:
He searches for the Web site in the Google search widget and then just uses the results page Chrome tab as his entire browser.
His default browser is not set to Chrome.

5 06:21 PM · Jul 15, 2021

Who would do this, you might ask? None other than Google's own Search App; you know the one that comes on every reputable Android device via the ubiquitous home screen search widget.

Known as the "Android Google Search App" ("AGSA", or "AGA"), this humble text input is the source of a truly shocking amount of web traffic; traffic that all goes to Chrome, no matter the user's choice of browser.

There were justifiable reasons to add code like this. Early in the life of the CCT protocol, before support was widespread, many browsers exhibited showstopper bugs. 2021 is far advanced from those early days, however, and so the primary effect of calling to Chrome is to distort the market for browsers and undermine user choice. This behaviour subverts user privacy, undermines the ecosystem benefits of engine diversity, and makes it hard for alternative browsers to compete on a level playing field.

This situation is admittedly better than the wholesale neutering of important developer-facing features by WebView IABs, but a Hobson's Choice none the less.

WebLayer: New Frontiers In User Confusion #

Google can (and should) revert to the system default of affirmatively respecting user choice in browsers by deleting the offending choice override. Given that AGSA uses CCT to load web pages rather than a WebView, this is nearly trivial today. CCT's core design is sound and has enormous potential if made mandatory in place of WebView IABs by the Android and Play teams.

There's reason to worry that this is unlikely.

Instead of addressing frequent developer requests for features in the CCT library, the Chrome for Android team has invested heavily in the "WebLayer" project. You can think of WebLayer like a WebView-with-batteries-included, repairing issues related to missing features but continuing to fracture state and user choice.

There is a positive case for WebLayer: as a replacement for WebViews in the context of browsers, it's a major step forward. In the context of IABs, however, WebLayer looks set to entrench user-hostile patterns further.

This subversion of choice extends a dispiriting trend in search apps that fancy themselves browsers without even attempting to earn a user's business as a browser.

In addition to Google Go, the Google app for iOS as well as Microsoft's Bing app for Android both capture outbound links in WebView IABs, subverting both browser choice and feature availability for developers. If there's any mercy, it's that their relatively lower use somewhat limits their impact on the overall ecosystem. Adopting WebLayer will not meaningfully improve the user experience or privacy of these amnesiac browsing experiences.

Google and Apple have the chance to lead, to show they aren't hostile to users, and remove the permission structure for lousy behaviour that less scrupulous players exploit. More on that in a moment.

iOS's Outsized, Malign Influence #

Imagine if automakers could only use one government-mandated engine model across all cars and trucks. Different tires and upholstery only go so far. If the engine is underpowered, many tasks might not be possible, rendering whole vehicle classes pointless. That's the situation iOS creates for browser makers and the browser-downloading public, and the only recourse it to buy a new phone.

iOS matters because wealthy users carry iPhones. It's really as simple as that. Even when Apple's products fail to gain a numerical majority of users in a market, the margin contribution of iOS users can dominate all other business considerations.

From at least 2012, Apple has deigned to allow "competing browsers" in its App Store. Those applications could not be browsers in any meaningful sense as they could not supplant Safari as the default handler of http/https links. The long charade of choice without effect finally ended with the release of iOS 14.2 in late 2020, bringing iOS into line with every other significant OS in supporting alternative browsers.^[10]

But Apple has taken explicit and extensive care to ensure that this choice is only ever skin deep on iOS. Browsers on Windows, Linux, ChromeOS, Android, and MacOS can be Integrated Browsers. iOS, meanwhile, restricts browsers to shells over the system-provided WebView.

Unlike WebView browsers on other OSes, Apple locks down these components in ways that prevent competition in additional areas, including restrictions on network stacks that block improved performance, new protocols, or increased privacy. These restrictions make some sense in the context of WebView IABs, but extending them to browsers only serves to deflect pressure from Apple to improve their browser.

Perhaps it would be reasonable for iOS to foreclose competition from integrated browsers and insist on uniquely constrained WebViews. Such policies would represent a different view of what computing should be if native apps were required to live within similar limits. However, Apple is happy to provide a much wider variety of features to unsafe native applications so long as they comply with the coercive terms of its App Store.

Integrated browsers present a model that, if allowed to flourish, pose a threat to the fundamentals of Apple and Google's whale-based, dopamine fueled, "casual" gaming monetisation rackets.

Unlike other native apps, integrated browsers are principally concerned with user safety. A safe-by-default, capable platform with low-friction discovery could obviate the core justification for app stores: that they're necessary to keep in check the over-powered-by-default apps built to an OS's proprietary platform.

Apple forestalls this bottom-line threat by keeping the web on iOS from gaining reasonable feature parity. Outlawing integrated browser choice leaves only Apple's own, farcially under-powered, Safari/WebKit browser/engine...and there's precious little that other WebView browsers can do to improve the situation at a deep level.^[11]

Web developers are understandably livid:

Ada Rose Cannon @AdaRoseCannon

Seeing a Web App I worked on used by *Apple* to justify that the Web is a viable platform on iOS is bullshit

The Web can be an ideal place to build apps but Apple is consistently dragging their heals on implementing the Web APIs that would allow them to compete with native apps twitter.com/stopsatgreen/status/1389593307219701760

692 06:05 PM · May 4, 2021

Ada Rose Cannon @AdaRoseCannon

In addition, by refusing to let any other Web browser engines run on iOS. They are preventing any other browser filling in the feature gap. Truly holding back Web Apps on iOS.

149 06:05 PM · May 4, 2021

Ada Rose Cannon @AdaRoseCannon

I have defended Apple's choice to restrict web browsers on their platform before and I still do but they can't have their cake and eat it to.

They should not hold back Web Apps with one hand and then turn around and say that Web Apps can compete with native apps.

153 06:10 PM · May 4, 2021

Pointing to a site of serial developer mistreatment to justify other developer-hostile App Store policies takes next-level chutzpah.

Developer anger only hints at the underlying structural rot. 25+ years of integrated browser competition has driven waves of security, capability, and performance improvements. Competition has been so effective in delivering these benefits that browsers now represent most computing time on OSes with meaningful and integrated browser choice.

Hollowing out browser choice while simultaneously starving Safari and WebKit of resources, somewhat miraculously, put the genie back in the bottle. Privacy, security, performance, and feature evolution all suffer when the competition is less vibrant — and that's how Apple likes it.

Mark(et)d Impacts #

A vexing issue for commentators regarding Apple's behaviour in this area is that of "market definition". What observers should understand is that, in the market for browsers, the costs that a browser vendor can inflict on web developers extend far beyond the market penetration for their specific product.

A typical (but misleading) shorthand for this is "standards compliance". While Apple's engine falls woefully short on support for ratified standards, that isn't even the beginning of the negative impacts.^[12] Because the web is an open, interoperable platform, web developers build sites to reach the vast majority of browsers from a single codebase.

When browsers with more than ~10% share fail to add a feature or exhibit nasty bugs, web developers must pay attention and work around these limitations. In the case of outright missing APIs, entire classes of content may simply be viewed as unworkable. The cost of these capability gaps is steep. When the web cannot deliver experiences that iOS native apps can (a very long list), businesses must build entirely different apps using Apple's proprietary tools. These apps, not coincidentally, can only be distributed via Apple's high-tax App Store.

A lack of meaningful user choice in browsers leads directly to higher costs for users and developers across the entire digital ecosystem even if they don't use Apple's products. The permission structure Apple's norm-eroding policies have constructed has served to justify some of the worst privacy and choice-undermining behaviour of tech giants. Apple's leadership in the race to the bottom has inspired a burgeoning field of fast-followers.

Beyond direct harms, interested parties should not consider browser choice as somehow orthogonal to other objectionable App Store policies; they are part and parcel of an architecture of control that tilts commerce into coercive, centralising App Stores. No matter how Apple wants to define the market, its actions distort and undermine competition.

Small Changes to Restore Choice #

Here's a quick summary of the systems and variations we've seen thus far, as well as their impacts on user choice:

Proposals to repair this profoundly broken situation must centre first on the effectiveness of browser choice. Some policymakers have suggested returning to browser choice ballots, however these will not be effective in a world where user choice is undermined no matter which browser they choose. Interventions to encourage informed browser choice cannot have a positive effect until the impact of choices can be assured.

Thankfully, repairing the integrity of browser choice in the mobile ecosystem can be accomplished with relatively small interventions. We only need to ensure that integrated browsers are universally available and that when third-party content is displayed, user choice of browser is respected.

Android #

Repairing the IAB situation will likely require multiple steps, given the extreme delay in new Android OS revisions gaining a foothold in the market. Thankfully, many fixes don't need OS updates:

• Google should update the CCT system to respect browser choice when loading third-party content and require updates to CCT-using apps to this new behaviour within six months.

  • Verification of first-party content for use with specific engines is possible thanks to the Digital Asset Links infrastructure that underpins Trusted Web Activities, the official mechanism for putting web apps in the Play Store.

• AGSA and Google Go should respect user choice via CCT.

• Android's WebView and WebLayer should be updated with code to detect a new HTTP header value sent with top-level documents that cause the URL to be opened in the user's default browser (or a CCT for that browser) instead.

  • These systems update out-of-band every six weeks on 90+% of devices, delivering quick relief.
  • As a straw-person design, the existing Content Security Policy system's frame-ancestors mechanism can be extended with a new 'system-default' value.
  • Such an opt-out mechanism preserves WebViews for first-party and second-party use-cases (those sites will simply not set the new header) while giving third-parties a fighting chance at being rendered in the user's default browser.
  • Apps that are themselves browsers (can be registered as default http/https handlers) would be exempt, preserving the ability to build WebView browsers. "Browserness" can be cheaply verified via an app's manifest.

• Google should provide access to all private APIs currently reserved to Chrome, including but not limited to the ability to install web applications to the system (a.k.a. "WebAPKs").

Future releases of Android should bolster these improvements by creating system-wide opt-out of WebView and WebLayer IABs.

Play policy enforcement of rules regarding CCT, WebView, and WebLayer respect for user and developer choice will also be necessary. Such enforcement is not challenging for Google, given its existing binary analysis infrastructure.

Together, these small changes can redress the worst anti-web, anti-user, anti-developer, and anti-choice behaviour of Google and Facebook regarding Android browsers, putting users back in control of their data and privacy along the way.

iOS #

iOS begins from a more troubling baseline but with somewhat better IAB policies. What's undermining user choice there require deeper, OS-level fixes, including:

• Integrated browser choice, including access to APIs that iOS restricts to Safari today, such as:

  • The ability to create sandboxed subprocesses for renderers.
  • Push Notifications APIs.
  • Adding web apps to the home screen, including PWA installation.
  • Support in Web.app for alternative engine runtimes to ensure that home screen shortcuts and PWAs run in the correct context.

• SFSafariViewController support for browsers other than Safari.

• All apps that load non-consenting third-party websites (outside of edge cases like authentication flows) in IABs should be required to update to SFSafariViewController,

• Apple's WebViews should support Content-Security-Policy: frame-ancestors 'system-default'

Allowing integrated browsers will require updates to Apple's App Store policies to clarify that alternative engines are permitted in the context of com.apple.developer.web-browser entitled applications.

For Markets To Work, Choice Must Matter #

The mobile web is a pale shadow of its potential because the vehicle of progress that has delivered consistent gains for two decades has silently been eroded to benefit native app platforms and developers. These attacks on the commons have at their core a shared disrespect for the sanctity of user choice, substituting the agenda of app and OS developers for mediation by a user's champion.

This power inversion has been as corrosive as it has been silent, but it is not too late. OSes and app developers that wish to take responsibility can start today to repair their own rotten, choice-undermining behaviour and put users back in control of their browsing, their data, and their digital lives.

The ball's in your court, platforms.